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DISCLAIMER 


This  publication  discusses  three  procedures  which  may  be 
utilized  in  the  experimental  design  -  data  analysis 
components  of  a  remote  access  computer  service  selection. 
It  is  not  intended  to  represent  the  entire  government 
computer  service  procurement  process.  Consideration  of 
testing  costs,  bidding  procedures,  environment  test 
conditions,  and  other  selection  issues  are  beyond  the  scope 
of  this  publication. 
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Application  of  Measurement  Criteria  in  the  Selection 
of  Interactive  Computer  Services 


Paul  D.  Amer 


ABSTRACT 

This  publication  emphasizes  the  data  analysis  component 
of  the  computer  service  selection  process.  A  computer 
service  selection  model,  introduced  in  NBS  Special 
Publication  500-44,  is  presented  and  three  binary  type 
selection  procedures  applicable  in  the  measurement  phases  of 
that  model  are  given.  A  binary  type  procedure  determines 
which  competing  computer  services  perform  above  and  which 
perform  below  a  specified  performance  level.  Those  services 
determined  to  perform  below  a  specified  performance  level 
can  be  eliminated  or  penalized  depending  on  whether  the 
performance  specification  is  mandatory  or  desirable.  The 
procedures  explicitly  specify  prior  to  measurement  an 
appropriate  decision  rule  and/or  the  number  of  test 
measurements  required  in  a  comparison  effort  to  attain  a 
given  level  of  statistical  confidence  in  the  results. 
Experimental  data  from  a  previous  case  study  are  reanalyzed 
to  illustrate  application  of  the  selection  procedures.  If 
the  collection  of  measurements  is  necessary  in  a  computer 
service  selection,  then  these  procedures  (or  similar 
statistically  sound  procedures)  should  be  employed  because 
of  their  mathematical  justification. 


Key  words:  Binary  selection;  computer  comparison; 
computer  performance  measurement;  computer  service;  data 
analysis;  experimental  design;  performance  evaluation; 
ranking  and  selection;     selection  methodology. 


1.0  INTRODUCTION 

NBS  Special  Publication  500-44  [MAM79a]  (herein 
referred  as  SP  500-44)  addresses  the  comparison  and 
selection  of  remote  access  interactive  computer  services  in 
terms  of  a  statistical  comparison  methodology.  That 
methodology  relies  principally  on  the  collection  and 
statistical  analysis  of  measurement  data  obtained  from  the 
interactions  between  a  computer  service  and     a     user.  Four 
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general  test  procedures  or  experimental  designs  are 
described  for  use  in  the  data  analysis  component  of  the 
comparison  and  selection  process. 

Three  of  the  procedures  in  SP  500-44  determine  the 
"best"  of  a  set  of  computer  services  where  best  is  defined 
by  a  single  performance  index  and  either  a  mean,  percentile 
or  proportion  statistic.  For  example,  one  of  the  procedures 
might  be  employed  to  determine  the  computer  service  with  the 
shortest  mean  response  time  for  all  text-editor 
transactions . 

The  fourth  procedure  in  SP  500-44  determines  a  subset 
of  computer  services  that  includes  all  services  which,  for  a 
particular  performance  index,  perform  at  or  above  a 
specified  "single  proportion"  performance  level.  For 
example,  the  procedure  will  determine  a  subset  of  computer 
services  that  includes  all  services  for  which  at  least  80% 
of  the  response  times  are  less  than  two  seconds.  This 
procedure  is  a  "binary"  or  "yes-no"  type  procedure  since  for 
each  service  a  decision  is  made  either  to  include  it  in  or 
to  exclude  it  from  the  selected  subset.  A  binary  type 
procedure  provides  an  analyst  with  reliable  performance 
information  about  which  services  perform  above  and  which 
services  perform  below  a  specified  performance  level. 

This  Special  Publication  also  emphasizes  the 
experimental  design  -  data  analysis  components  of  the 
computer  service  selection  process.  Introduced  are  two 
binary  type  selection  procedures  which  are  extensions  of  the 
binary  type  procedure  in  SP  500-44.  The  first  procedure 
determines  a  subset  of  computer  services  that  includes  all 
services  which,  for  a  particular  performance  index,  perform 
at  or  above  a  specified  "dual  proportion"  performance  level. 
For  example,  the  procedure  will  determine  a  subset  of 
computer  services  that  includes  all  services  for  which  at 
least  50%  of  the  response  times  are  less  than  two  seconds 
and  at  least  80%  of  the  responses  are  less  than  fWe 
seconds.  A  dual  proportion  permits  greater  flexibility  than 
a  single  proportion  in  defining  measurable  performance  by 
specifying  two,  rather  than  one,  statistical  values  of 
interest  of  each  service's  performance  distribution. 

The  second  procedure  determines  a  subset  of  computer 
services  that  includes  all  services  which,  for  a  particular 
performance  index,  perform  at  or  above  a  specified  "upper 
proportion"  performance  level  and  excludes  all  services 
which  perform  at  or  below  a  specified  "lower  proportion" 
performance     level.       For  example,  suppose  computer  services 
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are  defined  to  be  in  one  of  three  categories:  (1)  those 
with  80%  or  more  responses  less  than  two  seconds,  (2)  those 
with  50%-80%  responses  less  than  two  seconds,  and  (3)  those 
with  50%  or  less  responses  less  than  two  seconds.  This 
procedure  will  determine  a  subset  that  includes  all  services 
in  category  (1)  and  excludes  all  services  in  category  (3). 
An  upper-lower  proportion  combination  permits  greater 
control  than  a  single  or  dual  proportion  by  regulating  just 
how  poorly  a  service  can  perform  and  still  be  selected  due 
to  variability  present  in  a  measurement  study. 

An  important  characteristic  of  the  procedures  in 
SP  500-44  and  here  is  the  analyst's  ability  to  control 
(limit,'  bound)  the  probability  of  obtaining  an  incorrect 
selection  (further  explained  in  Section  3.2).  These 
procedures  explicitly  specify  prior  to  measurement  an 
appropriate  decision  rule  and/or  the  number  of  measurements 
required  in  a  computer  service  comparison  to  attain  a  given 
level  of  statistical  confidence  in  the  results.  Although 
each  procedure  compares  services  for  a  single  performance 
index,  together  they  may  be  employed  for  several  indices 
with  the  individual  results  then  integrated  in  a 
comprehensive  selection  methodology  (see  Section  2.3).  The 
procedures  specifically  are  applicable  in  a  comparison 
effort  whenever  a  selection  methodology  requires  binary 
information  about  a  set  of  competing  computer  services,  i.e. 
which  services  do  and  which  services  do  not  provide  a 
certain  level  of  performance. 

If  the  collection  of  measurements  is  necessary  in  a 
computer  service  selection,  then  these  procedures  (or 
similar  statistically  sound  procedures)  should  be  employed 
because  their  mathematical  foundation  permits  an  analyst  to 
justify  and  document  the  selection  making  process. 


2.0   SELECTION  OF  INTERACTIVE  COMPUTER  SERVICES 

Federal  Information  Processing  Guidelines  for  the 
Measurement  of  Remote  Access  Interactive  Computer  Service 
Response  Time  and  Turnaround  Time  [FIP57]  indicates  that 
when  two  or  more  computer  services  are  being  compared, 
statistics  and  statistical  techniques  should  be  employed  for 
the  description  and  analysis  of  the  measurements  which  are 
collected.  Further,  since  the  statistical  techniques  which 
are  employed  can  influence  the  results,  the  techniques  must 
be  clearly  specified  in  advance.     With  regard  to  the  kind  of 
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statistics  to  employ,  the  guidelines  more  specifically 
state : 

"Quite  often  the  distributions  of  response  time 
and  turnaround  time  are  not  known  (in  closed 
mathematical  form) .  Since  most  of  the  data 
distributions  which  have  been  collected  and 
analyzed  have  been  non-normal  (non-Gaussian) , 
Gaussian  statistical  descriptors  such  as  the  mean 
and  standard  deviation  are  not  appropriate.  It  is 
therefore  recommended  that  non-parametric 
statistical  descriptors  be  employed  when 
specifying  response  time  and  turnaround  time." 

Performance  data  in  a  computer  service  selection 
usually  are  collected  in  an  uncontrolled  environment.  That 
is,  an  analyst  exercises  little  or  no  control  over  the 
software,  hardware  or  external  workload  of  a  computer 
service  during  the  test  periods.  (+)  This  mode  of  testing 
is  in  accord  with  the  actual  mode  of  utilization  of  a 
computer  service  once  it  has  been  selected.  Users  do  not 
have,  nor  do  they  desire,  control  of  the  technical  aspects 
of  how  services  ..are  produced  or  how  they  are  delivered  when 
they  purchase  computer  services  (in  contrast  to  computer 
systems)  [ABR77b] .  However,  they  do  desire  a  guaranteed 
level  of  service  at  their  terminal  interface  and  that  is 
precisely  what  is  measured.  Due  to  the  many  uncontrolled 
variables  present  in  a  typical  computer  service  evaluation 
effort,  there  is  an  unknown  degree  of  variability  in  the 
collected  measurements.  Therefore,  it  is  necessary  to 
employ  valid  statistical  techniques  and  to  collect  multiple 
measurements  to  confidently  obtain  an  accurate  estimate  of  a 
computer  service's  performance.  This  publication  addresses 
the  questions  of  what  data  analysis  technique  and  how  many 
measurements  are  necessary  for  a  desired  level  of 
statistical  confidence  in  the  comparison  and  selection 
results. 

A  computer  service  selection  model  (CSSM)  is  presented 
in  SP  500-44.  Based  on  that  model,  a  methodology  is  given 
which  relies  principally  on  the  statistical  analysis  of 
measurement  data  obtained  from  the  interactions  between  a 
computer  service  and  a  user.     The    methodology  incorporates 


(+)  Complete  specification  of  the  testing  environment  is 
beyond  the  scope  of  this  publication. 
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confidence  statements  about  the  probabilities  of  correct 
selections  made  at  various  phases  of  a  computer  service 
selection  process.  The  technical  and  economic  feasibilities 
of  applying  the  selection  methodology  are  verified  in  a  case 
study  comparing  four  heterogenous  remote  access  time  sharing 
services.  (It  is  recommended  that  the  reader  study 
SP  500-44  for  a  global  view  of  the  selection  process.  Some 
portions  of  SP  500-44  have  been  summarized  in  this  SP  for 
completeness. ) 

This  publication  presents  selection  procedures  beyond 
those  previously  published  in  SP  500-44  for  use  in  the 
measurement  phases  of  a  computer  service  selection  when  the 
criteria  for  selection  are  measurable  and  are  expressed  in 
terms  of  proportion  statistics.  Specifically,  they  are 
applicable  when  the  analyst  wishes  to  know  only  if  a  service 
performs  above  or  below  a  specified  performance  level. 

One  of  the  four  experimental  procedures  that  are 
presented  in  SP  500-44  (Section  2.4.1)  is  repeated  here  to 
assist  the  reader  in  understanding  proportion  criteria  and 
to  facilitate  presentation  of  the  two  additional  procedures 
that  follow.  In  conjunction  with  SP  500-44,  this 
publication  furthers  the  development  of  a  statistical 
methodology  for  the  comparison  and  selection  of  interactive 
computer  services. 


2.1  Classification  of  Performance  Criteria 

The  computer  service  comparison  and  selection  process 
is  a  complicated  one,  based  on  various  performance  indices. 
Some  indices  are  measurable  (such  as  service  response  time) , 
and  some  are  not  (such  as  ease  of  service  use  and  coherence 
of  service  documentation) .  For  example,  evaluating  service 
documentation  and  the  amount  of  main  memory  does  not  require 
measurement  collecting  activity,  whereas  evaluating  service 
turnaround  time  and  response  time  clearly  requires 
measurement.  When  criteria  are  based  on  measurable  indices, 
comparison  requires  collecting  and  analyzing  relevant 
performance  measurements  from  the  various  computer  services 
being  considered. 

Performance  criteria  can  be  divided  further  into  those 
which  are  mandatory  and  those  which  are  desirable.  A 
mandatory  criterion  defines  a  service  requirement  which  must 
be  satisfied  by  the  computer  services  being  considered  for 
selection.  Failure  to  satisfy  even  one  mandatory  criterion 
results     in  a  service's  elimination.     Desirable  criteria,  on 
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the  other  hand,  are  those  which  are  not  absolute 
requirements  for  acceptance,  but  which  if  satisfied  would 
facilitate  processing  of  a  user's  workload.  Therefore,  if  a 
given  computer  service  does  not  fulfill  some  desirable 
criteria,  it  would  continue  to  be  considered  for  selection, 
but  depending  on  the  overall  selection  methodology  failure 
to  satisfy  each  desirable  criterion  would  perhaps  invoke 
some  associated  penalty  on  the  respective  service. 
(Likewise,  for  each  desirable  criterion  satisfied,  a  given 
service  could  be  compensated  with  an  associated  reward.) 

Based  on  the  two  characteristics  described  above, 
performance  criteria  can  be  classified  as:  Mandatory 
Nonmeasurable  (MN) ,  Mandatory  Measurable  (MM) ,  Desirable 
Nonmeasurable  (DN) ,  and  Desirable  Measurable  (DM) .  Examples 
of  each  class  of  criteria  are  provided  in  Table  1. 

There  are  a  large  number  of  performance  indices  which 
may  be  used  for  defining  comparison  criteria  in  a  computer 
service  selection.  Abrams  and  Treu  [ABR77b]  have  tabulated 
more  than  fifty  measures  which  may  be  used  to  describe  the 
demands  and  needs  of  a  user  at  an  interactive  terminal. 
Those  measures  which  describe  the  time,  length,  rate  and 
ratios  of  user-computer  interactive  behavior,  are  likely  to 
be  of  primary  interest  in  the  evaluation  and  selection  of 
computer  services,  especially  the  three  measures  of  response 
time,  turnaround  time  and  cost. 

Ultimately  a  decision  about  how  many  and  which 
performance  factors  are  most  important  in  an  evaluation  is  a 
management  policy  issue.  One  user  may  view  system 
reliablility  as  the  over-riding  consideration  in  a  service 
selection  effort  while  another  might  view  cost  and  response 
time  for  short  edit  commands  as  the  most  important  factors. 
Usually  some  combination  of  various  measures  of  service 
response  time  and  costs  is  used  in  the  selection  process. 
This  publication  makes  no  statement  about  which  performance 
indices  should  be  used  in  defining  selection  criteria.  It 
does,  however,  assume  that  selection  criteria  exist  and  can 
be  stated  in  advance  of  any  measurement  effort. 


2.2  Application  of  Performance  Criteria 

The  CSSM  in  SP  500-44  indicates  a  sequence  in  which  the 
four  classes  of  performance  criteria  typically  are  applied 
in  most  service  selection  methodologies.  This  model  is 
repeated  here  in  Figure  1.  Phase  I  involves  the  application 
of  MN  criteria.     The  phase     is    managed    easily    since  each 
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Table  1 


Examples  of  Performance  Criteria 

Type  Example 

Mandatory  1.     The  system  must  be  fully  delivered  and  operational 

Nonmeasurable  no  later  than  September  1,  19T9- 

2.     Timesharing  service  must  include  FORTRAN,  Basic,  Lisp, 
SNOBOL,  and  editing  facilities. 


Mandatory  1.     The  mean  turnaround  time  for  all  student  jobs  must 

Measurable  be  less  than  15  minutes. 

2.     95^  of  all  trivial  command  response  times  must  be 
less  than  1  second. 


Desirable  1.     It  is  desirable  that  the  system  include  Pascal  and 

Nonmeasurable  COBOL  facilities. 

2.     It  is  desired  that  the  system  provide  a  text  editing 
capability. 


Desirable  1.     It  is  desired  that  the  system  provide  a  mean 

Measurable  turnaround  time  for  the  benchmark  run  of  5  minutes 

or  less. 

2.     It  is  desired  that  95%  of  all  trivial  command  response 
times  be  0.5  seconds  or  less. 
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Figure  1  Computer  Service  Selection  Model  (CSSM) 
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computer  service  either  does  or  does  not  have  the  required 
characteristic.  All  of  those  services  which  do  not  satisfy 
the  MN  criteria  are  eliminated. 

Phase  II  involves  the  application  of  MM  criteria.  In 
general  for  each  MM  criterion,  performance  measurements  are 
gathered  from  every  computer  service  and  a  decision  is  made 
as  to  whether  or  not  the  criterion  is  satisfied.  Failure  to 
satisfy  a  single  MM  criterion  results  in  a  service's 
elimination.  A  typical  selection  criterion  that  might  be 
applied  in  phase  II  is  "consider  only  those  services  which 
provide  a  turnaround  time  of  five  minutes  or  less  for  at 
least  80%  of  the  remote  job  entry  requests." 

Finally,  determination  of  the  best  computer  service  is 
made  in  phase  III.  This  phase  is  separated  into  two  parts, 
phase  IIIA  for  the  application  of  DN  criteria  and  phase  IIIB 
for  the  application  of  DM  criteria.  (Note  that  it  is  not 
implied  that  DN  and  DM  criteria  are  necessarily  applied 
separately  or  in  any  particular  order.)  Phase  III  involves 
gathering  both  measurement  and  subjective  data  and 
integrating  all  of  the  information  in  a  complex  manner  to 
determine  which  computer  service  is  the  best  one.  A 
service's  failure  to  satisfy  a  desirable  criterion  results 
not  in  its  elimination,  but  rather  in  a  penalty  to 
compensate  it  with  those  services  that  do  satisfy  the 
desirable  criterion.  Different  selection  methodologies 
incorporate  these  penalties  into  the  overall  selection 
process  in  different  manners,  yet  the  effect  is  the  same. 
Typical  questions  asked  in  phase  III  are  "does  a  service 
provide  adequate  software  and  hardware  documentation?"  and 
"does  a  service  have  a  desired  x%  of  its  response  times  less 
than  a  five  second  threshold?". 

In  both  phases  II  and  IIIB,  comparison  requires 
collecting  and  analyzing  relevant  performance  measurements 
from  the  various  computer  services  under  consideration:  in 
phase  II,  to  eliminate  those  services  failing  to  satisfy 
mandatory  performance  requirements,  and  in  phase  IIIB,  to 
penalize  those  services  failing  to  perform  at  desirable 
performance  levels. 


2.3  Methodology  for  Selection 

To  understand  better  how  the  procedures  described  in 
Section  3  fit  into  the  overall  selection  process,  two 
general  approaches  to  computer  service  selection  are  now 
discussed.       The     first    approach  applies  a  set  of  mandatory 
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criteria  and  then  chooses  the  least  expensive  service 
remaining.  In  terms  of  the  CSSM,  phase  III  is  greatly 
simplified  with  this  approach  (since  minimum  cost  becomes 
the  only  desirable  criterion) .  When  possible  such  an 
approach  is  suggested  due  to  its  ease  and  objectivity.  Sole 
use  of  mandatory  criteria  to  define  computer  service 
requirements,  however,  is  appropriate  only  when  a  user  has  a 
well  justified  set  of  functional  (nonmeasurable)  and 
performance  (measurable)  requirements.  The  latter  occurs  if 
there  exist  threshold  levels  of  performance  for  which 
"better"  performance  represents  no  significant  improvement 
in  satisfying  processing  needs.  For  example,  in  a 
particular  application,  a  service  for  which  80%  of  its 
responses  occur  in  less  than  two  seconds  may  be  for 
practical  purposes  just  as  good  as  a  service  for  which  80% 
of  its  responses  occur  in  less  than  one  second.  That  is,  as 
long  as  80%  of  its  responses  occur  in  less  than  two  seconds, 
a  service  is  considered  acceptable.  Once  a  certain 
threshold  level  of  performance  is  assured,  other 
considerations  such  as  cost  and  available  software  become 
more  important  factors  in  the  selection  process. 

A  second  approach  for  use  in  more  complex  selections 
applies  a  set  of  mandatory  criteria  (thereby  reducing  the 
original  set  of  services  to  a  set  of  minimally  acceptable 
ones)  and  then  applies  a  set  of  desirable  criteria.  For 
each  desirable  criterion  satisfied,  a  service  is  rewarded  an 
amount  (either  in  dollars  or  points)  which  represents  the 
effort  required  for  the  user  to  make  up  for  the  service 
feature  had  it  been  lacking.  After  all  desirable  criteria 
are  applied,  the  service  with  the  lowest  cost  or  score  is 
selected.  This  technique  is  part  of  a  cost-value  selection 
methodology  detailed  in  [JOS771  and  summarized  in  [TIM73] . 
Other  selection  methodologies  define  a  weights  and  scores 
approach  [DOW67]  and  a  figure  of  merit  approach  [ABR77b] . 
The  reader  should  be  familiar  with  one  or  more  of  these 
techniques  to  understand  fully  how  the  procedures  described 
below  fit  into  the  overall  selection  process. 

Regardless  of  the  selection  methodology  used, 
information  about  the  computer  services  being  compared  with 
respect  to  particular  performance  indices  is  needed.  The 
procedures  presented  in  this  publication  determine  which 
services  perform  at  specified  performance  threshold  levels 
when  those  levels  are  expressed  in  terms  of  proportion 
statistics.  I^  the  procedures  are  applied  in  phase  11_  of  a 
selection  process ,  the  included  services  continue  to  be 
considered  for  selection,  while  the  excluded  services  are 
eliminated .       If  the  procedures  are  applied  in  phase  IIIB  of 
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a  selection  process ^  the  included  services  are  properly 
rewarded  and/or  the  excluded  services  are  properly 
penalized .  In  general,  the  procedures  are  employed  as  part 
of  a  binary  decision  making  process.  Each  procedure 
provides  the  analyst  with  a  confidence  statement  which 
characterizes  a  comparison  study's  probability  of  success. 
That  is,  confidence  statements  similar  to,  "we  are  90% 
confident  that  services  x,  y,  z  provide  a  specified  level  of 
performance  while  services  a,  b,  c  do  not"  can  be  made  after 
measurement. 

For  computer  service  measurement  analyses,  there  are 
four  steps  which  must  be  followed  to  collect  meaningful 
measurements  from  the  computer  services  being  compared. 
They  are: 

1.  determination  of  performance  criteria  which 
will  form  the  basis  of  a  service  comparison 
[ABR77b] , 

2.  development  of  a  user  scenario  that  is 
representative  of  a  projected  workload  [CR074, 
NOL74,  WAT77,  WRI76] , 

3.  translation  of  a  scenario  into  individual 
scripts  executable  on  the  respective  services 
under  test  [MAM79a] ,  and 

4.  collection  and  analysis  of  the  data  required 
for  a  comparison. 

The  procedures  now  presented  address  issues  relevant  to  step 
four;     some  references  for  the  others  are  provided. 


3.0  APPLICATION  OF  PROPORTION  CRITERIA 

The  application  of  proportion  criteria  in  a  computer 
service  selection  occurs  when  an  analyst  wants  to  know  which 
services  provide  a  minimum  level  of  performance.  The 
efficiency  of  the  data  collection  process  and  the  validity 
of  the  data  analysis  in  making  the  binary  decisions  required 
in  such  an  application  depend  upon  the  choice  of  an 
appropriate  experimental  design.  An  area  of  statistics 
entitled  "Statistical  Ranking  and  Selection"  provides  an 
appropriate  foundation  for  the  data  collection  and  analysis 
component  of  a  computer  service  comparison  and  selection 
(see  [DUD78]  or  [GIB77]  for  a  survey  of  ranking  and 
selection  techniques) . 
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The  statistical  ranking  and  selection  literature  refers 
to  the  binary  decision  making  process  as  selection  "better 
than  a  standard"  or  "subset  selection".  Reference  in  this 
publ ication  to  a  service  as  being  better  than  standard  is  an 
attempt  to  be  consistent  with  this  literature  and  does  not 
imply  a  "standard"  such  as  a  voluntary  national  standard  or 
a  Federal  Information  Processing  Standard .  Subset  selection 
procedures  based  on  a  proportion  statistic  are  appropriate 
for  the  application  of  measurable  criteria  in  a  computer 
service  selection.  Three  of  these  procedures  are  described 
here. 

The  actual  step-by-step  procedures  that  are  presented 
in  this  section,  combined  with  Tables  1-19  in  Appendix  B  of 
SP  500-44  and  Appendices  A  and  B  here,  are  complete  in  that 
they  may  applied  without  reference  to  other  books  or 
articles.  No  statistical  justification  for  the  procedures 
is  presented;  the  interested  reader  is  referred  to  [AME79] 
and  [AME80]  for  the  statistical  theory.  Other  references 
describing  the  application  of  ranking  and  selection  theory 
to  computer  selection  include   [AME78]   and   [MAM77] . 

To  those  readers  for  whom  this  is  a  first  exposure  to 
ranking  and  selection  procedures,  it  is  suggested  that 
Sections  3.2.1,  3.2.2  and  3.2.3  be  read  in  parallel  with 
Sections  4.2.1,  4.2.2  and  4.2.3,  respectively,  where  example 
applications  of  the  procedures  are  given.  This  mode  of 
reading  will  provide  an  intuitive  feel  for  the  procedures 
and  will  place  their  necessarily  abstract  and  general  steps 
in  an  environment  with  which  the  reader  is  already  familiar. 


3.1  Proportion  Criteria 

As  quoted  in  Section  2.0,  [FIP57]  recommends  the  use  of 
non-parametric  statistical  descriptors  when  specifying 
computer  service  response  time  and  turnaround  time.  In 
particular,  it  is  recommended  that  in  specifying  acceptable 
computer  service  response,  one  or  more  classes  of 
interactive  tasks  be  specified  to  the  degree  required  by  the 
complexity  of  the  workload.     The  examples  given  are: 

1.  In  the  use  of  the  text  editor  for  insertions, 
deletions  or  changes  involving  n  characters  or 
less,  the  response  time  shall  be  less  than  w 
seconds  50%  of  the  time. 

2.  In  the  compilation  of  PL/I  programs  of  less 
than    m    lines,  the  response  shall  be  x  seconds  or 
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less  90%  of  the  time. 


3.  Retrieval  requests  to  a  bibliographic 
information  retrieval  service,  in  which  p  criteria 
(keys)  shall  all  be  satisfied  in  order  for  an  item 
to  be  displayed,  shall  be  serviced  with  a  response 
time  of  u  seconds  or  less  95%  of  the  time. 

These  are  all  examples  of  "single"  proportion  criteria; 
each  one  defines  a  single  threshold  level  of  performance  (w 
seconds,  x  seconds,  u  seconds)  and  an  associated  minimum 
proportion  (50%,  90%,  95%).  (Note  that  percentages  such  as 
50%,  90%,  95%  are  equivalent  to  proportions  of  0.50,  0.90, 
0.95,  respectively.) 

It  also  is  possible  for  the  user  to  increase  the 
complexity  of  a  selection  criterion  (and  likewise  the 
complexity  of  the  associated  data  analysis)  by  using 
multiple  proportion  statistics  in  specifying  acceptable 
response.  For  instance,  number  1.  above  could  be  restated 
as: 

1'.  In  the  use  of  the  text  editor  for  insertions, 
deletions  or  changes  involving  n  characters  or 
less,  the  response  time  shall  be  less  than  w 
seconds  50%  of  the  time  and  less  than  2w  seconds 
85%  of  the  time. 

This  is  an  example  of  a  "dual"  proportion  criterion  since  it 
defines  two  threshold  levels  of  performance  (w  seconds,  2w 
seconds)  and  two  associated  minimum  proportions  (50%,  85%). 

Procedures  for  the  application  of  single  and  dual 
proportion  criteria  are  discussed  in  Sections  3.2.1  and 
3.2.2,  respectively.  A  procedure  for  the  application  of  an 
"upper-lower"  proportion  criterion,  a  variation  of  a  single 
proportion  criterion,  is  discussed  in  Section  3.2.3. 

A  basic  set  of  definitions  common  to  ranking  and 
selection  procedures  is  presented  in  Table  2  for  easy 
reference.  Table  2  should  be  read  in  conjunction  with  the 
next  section.  The  notation  is  consistent  with  the 
statistical  ranking  and  selection  literature  and  will 
facilitate  reference  to  that  literature. 
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Table  2 

Summary  of  Ranking  and  Selection  Terminology 


A.     Notation  common  to  all  procedures 


k: 
p*. 

n: 

service 


niomber  of  computer  services  in  the  study 

desired  level  of  confidence  in  the  correctness 
of  the  selection  results;  P*  is  a  probability  of 
correct  selection 

number  of  measurements  collected  from  each  computer 
service 


(i) 


the  ith  computer  service;  i  =  1,  2, 


B .     Notation  common  to  selection  based  on  single  proportion  criteria 


C(thld) : 
X(i): 

p(min) : 

p(i): 


measure  threshold  value  for  a  performance  index 

number  of  measurements  from  service(i)  which  are 
less  than  C(thld) 

minimum  proportion  of  measurements  less  than  C(thld) 
required  to  include  a  service  in  a  selected  subset 

true  proportion  of  measurements  from  service(i) 
which  are  less  than  C(thld) 


C.     Notation  common  to  selection  based  on  dual  proportion  criteria 


Cl(thld) : 
C2(thld) 

XIUJ,  X2(i): 
pl(min) ,  p2(min) 


pl(i),  p2(i) 


first  and  second  measure  threshold  values  for 
a  performance  index 

number  of  measurements  from  serviced)  which 
are  less  than  Cl(thld),  C2(thld),  respectively 

minimum  proportion  of  measurements  less  than  Cl(thld) 
C2(thld),  respectively,  required  to  include  a 
service  in  a  selected  subset 

true  proportion  of  measurements  from  service(i) 
which  are  less  than  Cl(thld),  and  greater  than 
or  equal  to  C2(thld) ,  respectively 


D.     Notation  common  to  selection  based  on  upper-lower  proportion  criteria 


C(thld) : 
X(i): 

p_upper (min) 
p_lower(max) 
p(i): 


measure  threshold  value  for  a  particular  index 

number  of  measurements  from  serviced)  which 
are  less  than  C(thld) 

minimum  proportion  of  measurements  less  than  C(thld) 
required  to  include  a  service  in  a  selected  subset 

maximum  proportion  of  measurements  less  than  C(thld) 
required  to  exclude  a  service  from  a  selected  subset 

true  proportion  of  measurements  from  service(i) 
which  are  less  than  C(thld) 
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3.2  Procedures  for  the  Application  of  Proportion  Criteria 


For  selection  based  on  proportions,  it  is  assumed  that 
there  are  k  >  0  computer  services  from  which  it  is  desired 
to  determine  a  subset  of  minimal  size  which  includes  all 
computer  services  satisfying  a  specific  performance 
requirement.  Each  procedure  achieves  a  correct  subset 
selection  with  probability  at  least  P*  (0  <  P*  <1) .  The 
value  P*  is  prespec if ied ;  its  complement  1-P*  is  the  upper 
bound  on  the  probability  of  an  incorrect  selection.  (That 
is,  the  selection  procedures  limit  the  probability  of  an 
incorrect  selection  to  no  more  than  1-P*.) 

For  each  procedure,  it  is  necessary  to  obtain  n 
independent  measurements  from  each  computer  service.  The 
requirement  that  the  data  be  statistically  independent  poses 
a  problem  that  must  be  given  careful  consideration.  If  the 
data  are  not  independent,  they  are  "correlated".  This  means 
that  the  value  of  a  current  measurement  is  related  to  or 
dependent  on  the  value  of  one  or  more  previous  measurements. 
In  a  comparison  study,  correlation  among  data  points  would 
exist  if  measurement  collection  did  not  account  for  the 
periodic  nature  of  the  workload  demands.  For  example,  if 
the  processing  requests  on  a  computer  service  significantly 
change  during  certain  periods  of  the  day,  consecutive 
measurements  taken  over  short  time  intervals  are  likely  to 
be  correlated.  That  is,  a  fast  (slow)  response  time  data 
point  is  likely  to  follow  a  fast  (slow)  response  time  data 
point  if  the  time  interval  between  collection  is  not  long 
enough  to  negate  the  periodic  effect. 

SP  500-44  contains  an  extensive  discussion  of  the 
statistical  notion  o£  correlation  and  how  it  relates  to 
computer  service  data  collection.  In  general,  if  data  are 
correlated,  the  number  of  measurements  needed  for  a  given 
probability  of  making  a  correct  selection  increases 
dramatically  since  correlated  measurements  provide  less 
"information"  than  do  independent  measurements.  There  are 
many  techniques,  however,  for  avoiding  correlation  in  the 
data,  such  as  properly  spacing  measurement  collection  in 
order  to  assume  independence.  With  careful  planning, 
adequate  time  over  which  to  spread  measurement,  and 
reasonably  well-behaved  computer  services,  a  correlation 
problem  often  can  be  avoided  without  violating  experimental 
objectives.  (+) 


(+)  In  some  cases  a  correlation  problem  is  unavoidable.  The 
author  is  developing  procedures  which  take  correlation  into 
account . 
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A  significant  advantage  of  these  procedures  is  that 
they  make  no  assumptions  about  the  underlying  distribution 
of  the  measurements .  That  is,  one  need  not  assume  that 
response  times,  turnaround  times,  etc.,  follow  a  normal, 
exponential,  or  any  other  often  assumed  distribution. 

The  goals  of  the  procedures  are: 

Section  3.2.1  -  Determining  S(single),  a  subset  of 
computer  services  that  includes  all  services 
better  than  a  single  proportion  criterion. 

Section  3.2.2  -  Determining  S(dual)  -  a  subset  of 
computer  services  that  includes  all  services 
better  than  a  dual  proportion  criterion. 

Section  3.2.3  -  Determining  S (upper-lower )  -  a 
subset  of  computer  services  that  includes  all 
services  better  than  an  upper  proportion  criterion 
and  excludes  all  services  worse  than  a  lower 
proportion  criterion. 


3.2.1  Selection  of  Services  Satisfying  a    Single  Proportion 
Criterion 

A  single  proportion  criterion  is  stated  in  terms  of  a 
measure  threshold  value,  C(thld),  and  a  minimum  proportion, 
p(min).  The  object  of  the  procedure  is  to  determine  a 
subset  of  the  k  computer  services  being  compared  which 
includes  every  service  whose  performance  distribution  (for  a 
particular  index)  has  values  less  than  C(thld)  at  least 
p(min)  proportion  of  the  time.  (+)  The  true  proportion  of 
values  from  the  ith  computer  service,  denoted  service(i), 
which  are  less  than  C(thld)   is  denoted  p(i) . 

For  example,  if  a  performance  index  of  interest  is 
turnaround  time  for  a  certain  sequence  of  user-computer 
interactions,  then  a  single  proportion  criterion  might  state 
that  at  least  eighty  percent  of  the  time  the  sequence  must 
be  executed  in  less  than  10  minutes.  In  this  case,  p(min)  = 
0.80  and  C(thld)   =  10  minutes. 


(+)  One  can  determine  a  subset  which  includes  every  computer 
service  whose  performance  distribution  has  values  greater 
than  C(thld)  at  least  p(min)  proportion  of  the  time  by 
replacing  "less"  (<)  with  "greater"  (>)  wherever  values  of 
C(thld)   are  referenced. 
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The  values  of  the  parameters  C(thld)  and  p(inin)  (and 
analogous  parameters  in  the  two  procedures  that  follow)  are 
analyst  choices  based  on  nonstatist ical  considerations  such 
as  past  performance  or  projected  performance  needs.  These 
values  also  may  be  the  results  of  psychological  studies  of 
human-computer  interaction  studies  which  indicate  threshold 
levels  of  computer  performance  necessary  for  efficient 
and/or  acceptable  human  performance  [CAR68,  MIL68,  WIL77] . 
For  example,  [MIL68]  indicates  that  most  response  delays  in 
conversational  interaction  between  human  and  information 
systems  must  be  less  than  15  seconds.  (In  this  case,  "most" 
might  be  interpretted  as  90%.) 

Selection  is  accomplished  by  collecting  measurements 
from  the  computer  services  under  study  and  estimating  each 
service's  p(i)  value.  Using  an  appropriate  decision  rule 
based  on  statistical  ranking  and  selection  theory,  the 
analyst  is  then  able  to  determine  those  computer  services 
which  satisfy  the  single  proportion  criterion. 

Denoting  S (single)  as  the  selected  subset  of  computer 
services,  by  definition  a  correct  selection  occurs  if  and 
only  if: 

V  i[p(i)    >     p(min)   — >  service(i)  e  S(single)] 

That  is,  for  each  and  every  service,  if  the  proportion  of 
values  less  than  a  threshold  is  greater  than  or  equal  to  a 
minimum  proportion  value,  then  that  service  is  included  in 
the  selected  subset   (see  Figure  2) . 

When  applying  proportion  criteria,  an  analyst  would 
prefer  a  procedure  which  perfectly  separated  all  "better 
than  (or  equal  to)  standard"  computer  services  from  all 
"worse  than  standard"  services.  With  only  a  finite  number 
of  measurements  from  which  to  draw  inference,  any  rule 
separating  "better"  services  from  "worse"  services  will  have 
positive  probability  of  two  kinds  of  selection  error:  that 
of  excluding  better  services,  and  that  of  including  worse 
services.  Based  on  practical  considerations  in  procurement 
situations,  it  is  assumed  better  to  minimize  the  chance  of 
the  latter  type  of  selection  error  (i.e.  it  is  better  to 
include  a  "worse"  service  than  to  omit  a  "better"  one) . 
This  attitude  is  justifiable  in  government  procurement  where 
efforts  must  be  made  to  minimize  the  chance  of  incorrectly 
claiming  a  vendor's  product  to  be  below  specifications. 
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EXAMPLE: 


SELECT  ALL  SERVICES  WHICH  HAVE  AT  LEAST  80% 
TURNAROUND  TIMES  LESS  THAN  10  MINUTES 


C(thld)  =  10  MINUTES  p(min)  =  0.80 


WORSE  THAN  STANDARD 


BETTER  THAN 
STANDARD 


0.0 


p(m  in) 


1.0 


BETTER 

BETTER 
WORSE 

WORSE 
BETTER  WORSE 


ALL  BETTERS 
(MAYBE  OTHERS) 


NO  BETTERS 


FIGURE  2:  APPLICATION  OF  A  SINGLE  PROPORTION  CRITERION 
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Therefore  this  procedure  determines  (with  probability 
at  least  P*)  a  subset  of  services  which  includes  all 
services  that  perform  better  than  C(thld)  at  least  p(min) 
proportion  of  the  time.  Other  assumptions  regarding  the  two 
kinds  of  selection  errors  result  in  different  subset 
selection  procedures;  they  are  not  investigated  here.  This 
selection  goal  is  extended  in  the  procedure  in  Section  3.2.3 
where  an  analyst  has  some  control  over  how  far  below 
standard  a  service  can  be,  and  still  be  included  in  the 
selected  subset. 

Note  that  for  a  subset  selection  policy,  there  is  also 
an  implicit  objective  of  minimizing  the  size  of  the  selected 
subset  (subject  to  a  1-P*  constraint  on  the  probability  of 
incorrect  selection) .  One  can  always  attain  a  correct 
subset  selection  with  probability  equal  to  one  by  including 
all  services  in  a  selected  subset  (since  such  a  subset 
indeed  would  include  all  better  than  standard  services) ,  yet 
this  contradicts  the  purpose  of  applying  proportion  criteria 
-  that  of  detecting  as  many  worse  than  standard  computer 
services  as  possible. 

The  procedure  is  a  four  step  process: 

Step  1:  Choose  appropriate  P*,  C(thld)  and  p(min) 
values  according  to  nonstatistical  considerations. 

Step  2:  Collect  n  independent  measurements  from  each 
of  the  k  computer  services.  The  analyst  chooses  n  based  on 
practical  considerations,  bearing  in  mind  that  as  n 
increases,  more  accurate  estimates  of  each  service's 
proportion  will  be  attained,  thereby  increasing  the  chances 
of  excluding  those  services  whose  performance  does  not 
satisfy  the  single  proportion  criterion. 

Step  3;  Let  X(i)  =  number  of  measurements  from 
service(i)  which  are  <  C(thld).  Since  n  is  identical  for 
each  computer  service,  the  X(i)  values  provide  information 
about  the  true  proportions. 

Step  A:  For  each  service,  compare  X(i)  to  a  constant 
M,  which  is  derived  from  statistical  ranking  and  selection 
theory.  If  X(i)  >  M,  then  include  service (i)  in  the 
selected  subset;  otherwise  exclude  it.  M  is  determined  by 
table  lookup  based  on  the  parameters  k,  n,  P*  and  p(min) . 
Extensive  values  for  M  are  tablulated  in  [AME79]  and  are 
reproduced  as  Tables  1-19  in  Appendix  B  of  SP  500-44.  (In 
these  tables  p(min)  is  referred  to  as  the  standard 
proportion.) 
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3.2.2  Selection  of  Services    Satisfying    a    Dual  Proportion 
Criterion 

A  dual  proportion  criterion  is  stated  in  terms  of  two 
measure  threshold  values,  Cl(thld)  and  C2(thld),  and  two 
minimum  proportions,  pl(min)  and  p2(min).  The  object  of  the 
procedure  is  to  determine  a  subset  of  the  k  computer 
services  being  compared  which  includes  every  service  whose 
performance  distribution  (for  a  particular  index)  has  values 
less  than  Cl(thld)  at  least  pl(min)  proportion  of  the  time 
and  has  values  less  than  C2(thld)  at  least  p2(min) 
proportion  of  the  time.  (+)  (Assuming  without  loss  of 
generalization  that  Cl(thld)  <  C2(thld),  then  to  be 
meaningful  pl(min)  <  p2(min).)  The  latter  part  of  a  dual 
proportion  criterion  equivalently  can  be  stated  as  "has 
values  greater  than  or  equal  to  C2(thld)  at  most  l-p2(min) 
proportion  of  time"    (see  Figure  3) . 

For  example,  if  a  performance  index  of  interest  is 
response  time  for  program  compilations,  then  a  dual 
proportion  criterion  might  state  that  at  least  60%  of  the 
compilations  must  take  less  than  thirty  seconds  and  at  least 
90%  of  the  compilations  must  take  less  than  one  minute.  In 
this  case  pl(min)  =  0.60,  p2 (min)  =  0.90,  Cl(thld)  =  30 
seconds  and  C2(thld)   =  1  minute. 

The  values  of  the  parameters  Cl(thld),  C2(thld), 
pi (min),  and  p2(min)  are  analyst  choices  based  on 
nonstatistical  considerations.  Selection  is  accomplished  by 
collecting  measurements  from  each  computer  service  under 
study  and  estimating  the  true  proportions  of  measurements 
from  service(i)   lying  below  Cl(thld)   and  C2(thld). 

Performance  measurements  from  the  computer  services 
fall  into  one  of  three  intervals;  interval  1  = 
[0,C1 (thld) ) ,  interval  3  =  [CI (thld) ,C2 (thld) )  and  interval 
2  =  [C2(thld),  infinity).  (Note  that  the  intervals  are 
ordered  1,3,2  and  not  1,2,3.)  The  true  proportions  of 
measurements  from  service (i)  which  lie  in  the  intervals 
1,3,2  are  denoted  pl(i),  1-pl ( i ) -p2 ( i ) ,  and  p2(i), 
respectively  (see  Figure  4)  . 


(+)  Justification  for  determining  a  subset  which  includes 
all  better  than  standard  services  rather  than  one  which 
precisely  separates  the  better  than  standard  services  from 
the  worse  than  standards  services  is  discussed  in  Section 
3.2.1. 
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1.0 


INTERVAL!     |        INTERVALS        |     INTERVAL  2 
Cl(thld)  C2(thld) 

FIGURE  4:  TYPICAL  CUMULATIVE  DISTRIBUTION 
FUNCTION  F(i)  OF  OBSERVATIONS  FROM  SERVICE(i) 
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A  dual  proportion  criterion  defines  service(i)  to  be 
better  than  standard  if  pl(i)  >  pl(min)  and  l-p2(i)  > 
p2(min).  Stated  in  other  terms,  pl(min)  is  a  minimum 
probability  of  a  measurement  lying  in  interval  1,  and 
p2(min)  is  a  minimum  probability  of  a  measurement  lying  in 
the  union  of  intervals  1  and  3.  (Note  that  the  latter  part 
of  this  statement  is  not  equivalent  to  saying 
p2 (min) -pi (min)  is  a  minimum  probability  of  a  measurement 
lying  in  interval  3.) 

Denoting  S(dual)  as  the  selected  subset  of  computer 
services,  by  definition  a  correct  selection  occurs  if  and 
only  if: 

Vi[{pl(i)   >     pl(min)   and  l-p2(i)   >  p2(min)} 
— >  service(i)  csjdual)] 

That  is,  for  each  and  every  service,  if  the  proportions  of 
values  less  than  both  threshold  values  are  greater  than  or 
equal  to  two  minimum  proportion  values,  respectively,  then 
that  service  is  included  in  the  selected  subset  (see  Figure 
5)  . 

A  dual  proportion  criterion  permits  greater  flexibility 
than  a  single  proportion  criterion  in  defining  performance 
requirements  in  a  computer  service  comparison  effort  by 
specifying  two,  rather  than  one,  statistical  values  of 
interest  of  a  performance  distribution.  The  added 
flexibility  of  a  dual  criterion  is  attained  at  the  expense 
of  a  more  complex  experimental  design. 

The  procedure  is  a  four  step  process: 

Step  1:  Choose  appropriate  P*,  Cl(thld),  C2(thld), 
pi (min)  and  p2 (min)  values  according  to  nonstatistical 
considerations . 

Step  2:  Collect  n  independent  measurements  from  each 
of  the  k  computer  services.  The  analyst  chooses  n  based  on 
practical  considerations,  bearing  in  mind  that  as  n 
increases  more  accurate  estimates  of  each  service's 
proportions  will  be  attained,  thereby  increasing  the  chances 
of  excluding  those  services  whose  performance  does  not 
satisfy  the  dual  proportion  criterion. 

Step  3:  Let  XI (i)  and  X2(i)  =  number  of  measurements 
from  service (i)  which  are  <  Cl(thld)  and  <  C2(thld), 
respectively.  Since  n  is  identical  for  each  computer 
service,     the    XI (i)     and    X2(i)     values  provide  information 
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EXAMPLE:  S  ELECT  ALL  SERVICES  WHICH  HAVE  AT  LEAST 

60%  RESPONSE  TIMES  LESS  THAN  30  SECONDS 
&  AT  LEAST  90%  RESPONSE  TIMES 
LESS  T  HAN  60  S  ECONDS 


Cl(thld)=  30  SECONDS 
C2(thld)  =  60  SECONDS 


pl(  min)=0.60 
p2(  min)=0.90 


WORSE  THAN  STANDARD  1 


BETTER  THAN  STANDARD  1 

—H 


0.0 


p  1  (min) 


1.0 


0.0 


WORSE  THAN  STANDARD  2 


BETTER  THAN 
STANDARD  2 

^-^^H^ 

p2(min)  1.0 


WORSE-WORSE 

BETTER-BETTER 
WORSE-BETTER 

BETTER-BETTER 
BETTER-WORSE 


ALL  BETTER-BETTERS 
(MAYBE  OTHERS) 


NO  BETTER-BETTERS 


FIGURE  5:  APPLICATION  OF  A  DUAL  PROPORTION  CRITERION 
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about  the  true  proportions. 

Step  For  each  service,  compare  XI (i)   and     X2(i)  to 

constants  Ml  and  M2 ,  respectively,  which  are  derived  from 
statistical  ranking  and  selection  theory.  If  XI (i)  >  Ml 
and  X2(i)  >  M2 ,  then  include  service(i)  in  the  selected 
subset;  otherwise  exclude  it.  Ml  and  M2  are  determined  by 
table  lookup  based  on  the  parameters  k,  n,  P*,  pl(min)  and 
p2(min).  Extensive  values  of  Ml  and  M2  are  tabulated  in 
[AME79]  and  are  reproduced  as  Tables  5-24  in  Appendix  A. 
(In  these  tables  pl(min)  and  p2 (min)  are  referred  to  as 
standard  proportions  1  and  2,  respectively.) 


3.2.3  Selection    of    Services    Satisfying      an  Upper-Lower 
Proportion  Criterion 

This  procedure  is  a  variation  of  the  procedure 
described  in  Section  3.2.1.  In  that  section,  it  is  assumed 
that  there  are  k  >  0  computer  services  being  compared  from 
which  a  subset  is  determined  that  includes  all  services 
satisfying  a  specific  single  proportion  criterion.  Such  a 
procedure  does  not  control  how  far  below  standard  a  computer 
service  can  perform  and  still  be  included  in  the  selected 
subset.  (Although  the  farther  below  standard  a  service 
performs  in  general,  the  greater  the  probability  that  it 
will  be  excluded  as  a  result  of  a  particular  empirical 
study)  The  procedure  only  achieves  with  confidence  P*  that 
all  better  than  standard  services  are  included.  Therefore 
it  is  possible  to  obtain  a  correct  selection  and  include 
several  worse  than  standard  services. 

For  this  procedure  it  is  assumed  that  there  are  k  >  0 
computer  services  from  which  a  subset  is  determined  that 
includes  all  services  better  than  an  "upper"  proportion 
criterion  and  that  excludes  all  services  worse  than  a 
"lower"  proportion  criterion.  An  upper-lower  proportion 
criterion  is  stated  in  terms  of  a  measure  threshold  value, 
C(thld),  and  minimum  upper  and  maximum  lower  proportions, 
p_upper (min)  and  p_lower (max) .  The  object  of  this  procedure 
is  to  determine  a  subset  1)  that  includes  every  computer 
service  whose  performance  distribution  (for  a  particular 
index)  has  values  less  than  C(thld)  at  least  p_upper (min) 
proportion  of  the  time  and  2)  that  excludes  every  service 
whose  performance  distribution  (for  the  same  index)  has 
values  less  than  C(thld)  at  most  p_lower(max)  proportion  of 
the  time.  (Note  that  if  p_upper (min)  =  p_lower (max) ,  the 
problem  reduces  to  perfectly  separating  better  services  from 
worse     services     for     a     single    proportion    criterion.  As 
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discussed  in  Section  3.2.1,  with  only  a  finite  number  of 
measurements  to  estimate  the  true  proportions,  perfect 
separation  is  not  possible.) 

For  example,  if  a  performance  index  of  interest  is  edit 
command  response  time,  then  a  performance  criterion  might 
state  that  at  least  seventy  percent  of  the  response  times 
must  be  less  than  three  seconds.  Using  the  procedure  in 
Section  3.2.1,  a  comparison  study  could  determine  a  subset 
of  services  which  includes  all  services  for  which  at  least 
70%  of  the  responses  are  less  than  three  seconds.  Using  the 
procedure  below,  an  analyst  could  design  the  same  study  also 
to  exclude,  with  probability  P*,  all  services  whose 
percentage  (proportion)  was  less  than  50%  (0.50).  In  this 
case,  p_upper (min)  =  0.70,  p_lower (max)  =  0.50  and  C(thld)  = 
3  seconds. 

The  values  of  the  parameters  C(thld),  p_upper(min)  and 
p_lower(max)  are  analyst  choices  based  on  nonstatistical 
considerations.  Service  selection  is  accomplished  by 
collecting  measurements  from  the  computer  services  under 
study  and  estimating  each  one's  p(i)  value. 

Denoting  S (upper-lower )  as  the  selected  subset  of 
computer  services,  by  definition  a  correct  selection  occurs 
if  and  only  if: 

Vi[p(i)   >     p_upper (min)   — >  service(i)  e  S (upper-lower ) 

and 

P(i)  £    p_lower(max)  — >  service(i)  /  S (upper-lower ) ] 

That  is,  for  each  and  every  service,  if  p(i)  is  greater  than 
or  equal  to  a  minimum  upper  proportion  value,  then  that 
service  is  included  in  the  selected  subset.  And  if  p(i)  is 
less  than  or  equal  to  a  maximum  lower  proportion  value,  then 
that  service  is  excluded  from  the  selected  subset.  No 
statement  is  made  about  those  services  for  which 
p_upper (min)   >  p(i)   >  p_lower(max)    (see  Figure  6). 

An  upper-lower  proportion  criterion  provides  more 
control  than  a  single  proportion  criterion  does  in  applying 
proportion  criteria  in  the  selection  process.  It  does  so  by 
permitting  an  analyst  to  regulate  how  poorly  a  service  can 
perform  and  still  be  selected  due  to  variability  in  the 
collected  measurements. 

An  upper-lower  proportion  criterion  differs  from  a 
single  or  dual  proportion  criterion  in  that  for  the  latter 
two  types  of  criteria,  the  analyst  chooses  n  bearing  in  mind 
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EXAMPLE:  SELECT  ALL  SERVICES  WHICH  HAVE  AT  LEAST  70% 
RESPONSE  TIMES  LESS  THAN  3  SECONDS  ALSO 
GUARANTEEING  TO  EXCLUDE  ALL  SERVICES  WITH 
AT  MOST  50%  RESPONSE  TIMES  LESS  THAN  3  SECONDS. 


C{thld)=  3  SECONDS  p -upper  (min)  =  0. 70 

p-lower(max)  =  0.50 


WORSE  THAN  BETTER  THAN 

LOV/ER  STANDARD  ?  UPPER  STANDARD 


r  II  I    II  I — h 


0.0  p- lower  (max)  p- upper  (min)  1.0 


BETTER 
BETTER 
? 

WORSE 


? 

BEHER 
WORSE 


INCLUDE 


EXCLUDE 


ALL  BETTERS 
NO  WORSES 
(MAYBE  ?'S) 


NO  BETTERS 
ALL  WORSES 
(MAYBE  ?  S) 


FIGURE  6:  APPLICATION  OF  AN  UPPER-LOWER 
PROPORTION  CRITERION 
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only  practical  considerations  and  a  desire  to  detect  as  many 
worse  than  standard  services  as  possible.  Applying  an 
upper-lower  criterion,  however,  requires  a  minimum  number  of 
measurements  which  must  be  collected  for  a  given  combination 
of  parameter  values.  This  is  because  n  must  be  large  enough 
to  differentiate  at  the  desired  confidence  level  between 
those  computer  services  better  than  the  upper  standard 
proportion  and  those  services  worse  than  the  lower  standard 
proportion . 


The  procedure  is  a  four  step  process: 


Step  1:  Choose  appropriate  P*,  C(thld),  p_upper (min) 
and  p_lower (max)  values  according  to  nonstatistical 
considerations . 


Step  2:  Collect  n  independent  measurements  from  each 
of  the  k  computer  services.  The  analyst  chooses  n  based  on 
table  lookup  in  Appendix  B. 

Step  3:  Let  X(i)  =  number  of  measurements  from 
service(i)  which  are  <  C(thld).  Since  n  is  identical  for 
each  computer  service,  the  X(i)  values  provide  information 
about  the  true  proportions. 


Step  £:  For  each  service,  compare  X(i)  to  a  constant 
M,  which  is  derived  from  statistical  ranking  and  selection 
theory.  If  X(i)  >  M,  then  include  service(i)  in  the 
selected  subset;  otherwise  exclude  it.  M  is  determined  by 
table  lookup  based  on  the  parameter  values  k,  n,  P*, 
p_upper (min)  and  p_lower (max) .  Extensive  values  for  M  are 
tabulated  in  [AME79]  and  are  reproduced  as  Tables  25-44  in 
Appendix  B.  (In  these  tables  p_upper(min)  and  p_lower(max) 
are  referred  to  as  upper  and  lower  standard  proportions, 
respectively. ) 

Table  entries  in  Appendix  B  are  computed  as  (n,M)  pairs 
such  that  if  n  measurements  are  collected  in  Step  2,  a 
decision  rule  will  exist  in  the  form  stated  in  Step  4. 
Three  (n,M)  pairs  are  computed  for  each  combination  of  k, 
P*,  p_lower (max)  and  p_upper (min) .  The  first  pair  indicates 
the  minimum  n  for  which  a  decision  rule  that  satisfies  the 
P*  requirement  exists.  The  next  two  pairs  are  for  larger  n 
in  the  situation  when  an  analyst  can  afford  more  data  and 
wants  the  same  P*  value.  Larger  n  values  result  in  more 
accurate  estimates  of  p(i)  and  therefore  in  a  more  accurate 
selection  process. 
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4.0  COMPARISON  RESULTS 


Using  the  procedures  above,  this  section  reanalyzes 
computer  service  measurement  data  collected  in  a  case  study 
performed  at  the  National  Bureau  of  Standards  during  Summer 
1978.  The  purpose  of  reanalyzing  past  data  is  to  illustrate 
the  selection  procedures  in  terms  of  an  actual  comparison 
effort.  The  original  study,  discussed  in  detail  in 
SP  500-44,  was  conducted  to  evaluate  the  time  and  cost 
required  to  apply  various  statistical  ranking  and  selection 
procedures  in  an  actual  computer  service  procurement 
environment.  Case  study  results  in  terms  of  the 
theoretical,  technical  and  economic  feasibility  of  comparing 
interactive  computer  services  can  be  found  in  [MAM79b] . 


4.1  Summary  of  a  Case  Study 

Four  heterogeneous  remote  access  time  sharing  services 
were  compared:  a  DEC  System-10  running  a  TOPS-10  Monitor,  a 
Honeywell  6180  running  MULTICS,  an  IBM  360/65  running 
MVT/TSO,  and  a  UNIVAC  1108  running  Exec  8.  A  "scenario",  or 
functional  description  of  an  interactive  benchmark  which  is 
to  be  executed  and  measured  on  each  computer  service  being 
compared  [WAT77] ,  was  designed  to  be  reasonably  typical  of 
the  functional  requirements  of  a  real  workload.  It  was  not 
expected  nor  supposed  to  be  representative  of  a  specific 
user's  workload;  rather  the  scenario  was  designed  to  be  a 
realistic  example  which  could  be  used  to  investigate  the 
overall  selection  methodology.  The  scenario  is  presented  in 
Table  3. 

The  scenario  was  translated  into  "equivalent"  scripts 
executable  on  the  four  services.  Eight  performance  indices 
based  on  the  activities  in  the  script  were  chosen  for 
computer  service  comparison  in  the  case  study.     They  are: 

1.  cost, 

2.  turnaround     time      for      the      entire  script 
execution, 

3.  response  time  for  the  bibliographic  retrieval 
(no.     2  in  the  scenario) , 

4.  response  time  for  the  FORTRAN  program  (no.  3 


Note:  Reference  to  commercial  products  in  this  report  is 
for  identification  only  and  does  not  imply  endorsement  by 
the  National  Bureau  of  Standards. 
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Table  3 
Scenario  for  the  Case  Study 


General  Specifications;  Think  time  (response-to-stimulus 
3elay)    Fo'r  all  user  commands  is  6  seconds. 

File  Character i sties ;  the  files  listed  below  are  to  be 
resident  on  the  respective  systems  before  the  start  of 
script  execution. 


Files     for     executing       a      COBOL  search 
bibliographic  database: 

SELECT:  compiled  COBOL  program 

BIB:  bibliographic  database 

ACCESS:  bibliographic  entries  to  be  found 

CHOSEN:  list  of  entries  retrieved 

RESULT:  summary  of  search 


of 


Files  for  executing  a  FORTRAN  version  of  the  BELL 
benchmark : 

BELL:         compiled  FORTRAN  program 

BELLIN:  input  specifications  for  the  program  run 
BELLOUT:  output  of  program  run 

File  for  interactive  FORTRAN  program,  INTERA,  a 
source  program  (with  errors)  to  compute  prime 
numbers 


Functional  Scr Ipt : 

1.  Logon  to  the  system. 

2.  Execute  SELECT. 

3.  Execute  BELL. 

4.  Copy  file  INTERA  to  file  INTRl . 

5.  Edit  INTRl,  correcting  a  syntax  error  by  changing  line 
14  to  read:     GO  TO  10. 

6.  Edit  INTRl,  correcting  a  logical  error  by  changing  line 
23  to  read:     PP=PP+1 . 

7.  Compile  INTRl. 

8.  Execute     INTRl.       This    will     initiate     the  following 
dialogue : 

ARE  YOU  TESTING  A  NUMBER?    (0   or  1) 
Enter :   @  (CR) 
ARE  YOU  GENERATING  PRIMES?    (0  or  1) 
Enter:   1  (CR) 
LIMIT  UNDER  WHICH  PRIMES  ARE ~T0  BE  GENERATED: 
Enter:   10^  (CR) 
25,   2,   3,   5,   7,   11,   13,   17,   19,   23,   31,   37,   41,   43,  47, 
53,   59,   71,   68,   71,   73,   79,   83,   89,  97 

ARE  YOU  TESTING  A  NUMBER?    (0  or  1) 
Enter :   0  (CR) 
ARE  YOU  GENERATING  PRIMES?    (0  or  1) 
Enter:  0  (CR) 
DO  YOU  WANT  TO  QUIT?    (0  or  1) 
Enter:   1  (CR) 

9.  Delete  INTRl  file. 
10.     Logoff  system. 
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in  the  scenario) , 

5.  response  time  for  the  copy  command  (no.  4  in 
the  scenario) , 

6.  response  time  for  the  first  edit  command  (no. 

5  in  the  scenario) , 

7.  response  time  for  the  second  edit  command  (no. 

6  in  the  scenario),  and 

8.  response  time  for  the  interactive  calculation 
of  all  prime  numbers  less  than  100  (no.  8  in 
the  scenario) . 

The  hardware/software  configuration  for  the  case  study 
is  illustrated  in  Figure  7.  The  scripts  were  repeatedly 
executed  on  each  of  the  computer  services  under  the 
automatic  control  of  a  remote  terminal  emulator  called  the 
Network  Access  Machine  [ROS75] .  Turnaround  time,  response 
time  and  cost  data  were  automatically  collected  for  each 
execution  of  each  script  by  a  minicomputer  called  the 
Network  Measurement  Machine  [ABR77a,  ROS76] .  The  data  were 
analyzed  using  a  statistical  package  called  OMNITAB  [H0G71] . 
Correlation  coefficients,  means,  percentiles  and  proportions 
were  calculated. 

In  the  case  "^study  comparison  results,  the  four  computer 
services  are  referred  to  as  service  1,  2,  3  and  4.  The 
numbers  were  randomly  assigned  to  the  four  services  to 
ensure  their  anonymity.  The  selection  results  discussed 
below  are  primarily  a  function  of  the  load  on  a  given  system 
and  not  of  the  individual  hardware/software  combinations 
providing  the  computer  service.  Thus,  it  cannot  be 
concluded  from  this  study  that  a  given  computer  system  is 
better  than  the  others,  but  only  that  a  given  computer 
service  is  better.  The  results  of  the  case  study  are  not  to 
be  construed  as  an  endorsement  erf  any  one  computer  service. 


4.2    Application  of  Proportion  Criteria 

The  purpose  of  the  case  study  was  to  demonstrate 
several  selection  procedures  each  of  which  might  require  a 
different  number  of  computer  service  measurements. 
Therefore  a  maximum  number  anticipated  for  any  selection 
procedure  was  collected.  In  a  real  application  of 
performance  criteria  using  ranking  and  selection  procedures, 
the  number  of  measurements  as  well  as  the  selection  criteria 
are  determined  in  advance.  Initially  the  scripts  were 
executed  120  times  on  each  service,  resulting  in  120 
measurements  for  each  of  the  eight  performance  indices 
listed  in  Section  4.1.     Due  to  the  independence  requirement. 
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each  data  set  was  eventually  reduced  to  90  (independent) 
measurements  for  each  index.  (See  SP  500-44:  Appendix  A 
and  Section  3.2.2  for  a  discussion  of  reduction  techniques.) 

In  the  examples  that  follow,  proportion  criteria  which 
likely  would  be  used  in  an  actual  procurement  are  chosen. 
Each  example  employs  one  of  the  experimental  procedures 
presented  in  Section  3.2.  (Additional  examples  can  be  found 
in  [AME79]  and  [MAM79a].)  In  those  examples  requiring  n  <  90 
measurements  from  each  computer  service  for  a  given  level  of 
confidence,  the  first  n  measurements  were  chosen.  In  a  real 
comparison,  when  application  of  one  criterion  requires  fewer 
measurements  than  are  collected  to  satisfactorily  apply 
another  criterion,  it  is  statistically  acceptable  (and,  in 
fact,  strongly  advisable)  to  use  more  than  the  required 
number  of  measurements.  The  examples  use  the  minimum  number 
of  measurements  for  purposes  of  clarity  only. 

In  the  examples,  the  values  of  those  parameters  which 
are  analyst  choices  were  specified  to  make  the  examples  as 
clear  and  explicit  as  possible.  Efforts  also  were  made  to 
select  parameter  values  that  are  likely  to  be  chosen  in  a 
computer  service  selection  process.  The  steps  in  the 
examples  parallel  the  steps  in  the  original  presentations  of 
the  procedures  in  Sections  3.2.1,  3.2.2  and  3.2.3. 

Table  4  presents  various  summary  statistics  for  the 
four  computer  services.  Only  the  proportion  information 
(i.e.  the  column  titled  "No.  Measurements  <  C(thld)")  is 
used  in  the  examples  below.  The  mean  and  quantile 
statistics  are  presented  to  provide  a  more  detailed 
comparison  picture.  The  complete  set  of  n  =  90  independent 
measurements  were  used  for  the  calculation  of  all  statistics 
in  Table  4  except  where  a  different  n  value  is  indicated. 


4.2.1  Selection  of  Services  Satisfying  a    Single  Proportion 
Criterion   (2  examples) 

Example 
(parallels  Section  3.2.1) 

Step  1:  Suppose  it  is  a  mandatory  criterion  that  a 
computer  service  be  able  to  interactively  calculate  the 
prime  numbers  between  1  and  100  (no.  8  in  Table  3)  in  less 
than  1.00  second  at  least  80%  of  the  time.  Further,  suppose 
it  is  desired  to  select  a  subset  which  includes  all  such 
services    with    a    probability  of  correct  selection  at  least 
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0.90.  Then  k  =  4 ,  P*  =  0.90,  C(thld)  =  1.00  and  p(min)  = 
0.80. 

Step  2:  Choose  n  =  90.  Ninety  independent 
measurements  have  been  made  on  each  computer  service. 

Step  3:  For  the  computer  services  under  study,  Table  4 
indicates  that  X(l)  =  49,  X(2)  =  90,  X(3)  =  74  and  X(4)  = 
32. 

Step  4^:  Table  3  in  Appendix  B  of  SP  500-44  shows  that 
M  =  64  for  this  example.  Therefore  include  services  2  and  3 
and  exclude  services  1  and  4,  i.e.  S(single)  =  {service(2), 
service (3)}.  Since  the  criterion  was  mandatory,  services  1 
and  4  would  be  eliminated  from  the  selection  process. 


Example 
(parallels  Section  3.2.1) 

Step  1:  Suppose  it  is  a  desirable  criterion  that  a 
computer  service  be  able  to  process  the  first  edit  command 
(no.  5  in  Table  3)  in  less  than  3.00  seconds  at  least  70%  of 
the  time.  So,  k  =  4,  C(thld)  =  3.00  and  p(min)  =  0.70.  Let 
P*  =  0.95. 

Step  2:  Choose  n  =  90.  Ninety  independent 
measurements  have  been  made  on  each  computer  service. 

Step  3^:  For  the  four  computer  services.  Table  4 
indicates  that  X(l)  =  84,  X(2)  =  90,  X(3)  =  50  and  X(4)  = 
57. 

Step  £:  Table  3  in  Appendix  B  of  SP  500-44  shows  that 
M  =  53  for  this  example.  Therefore,  include  services  1,  2 
and  4  and  exclude  service  3,  i.e.  S(single)  =  {service(l), 
service(2),  service(4)}.  Since  the  criterion  was  desirable, 
services  1,  2  and  4  would  be  appropriately  rewarded. 

Note  that  if  ranking  and  selection  techniques  were  not 
being  applied  in  this  example,  then  service  4  might  have 
been  excluded  since  the  "expected"  number  of  responses  less 
than  3.00  seconds  for  a  service  providing  minimally 
acceptable  response  is  70%  of  90  or  63,  and  X(4)  <  63.  But 
the  ranking  and  selection  procedure  indicates  that  because 
of  experimental  variation  service  4  cannot  be  excluded  at 
the  95%  confidence  level. 
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4.2.2  Selection  of  Services    Satisfying    a    Dual  Proportion 
Criterion  (1  example) 

Example 
(parallels  Section  3.2.2) 

Step  1:  Suppose  it  is  a  desirable  criterion  that  a 
computer  service  be  able  to  perform  the  bibliographic 
retrieval  (no.  2  in  Table  3)  in  less  than  8.0  seconds  at 
least  40%  of  the  time  and  in  less  than  10.0  seconds  at  least 
70%  of  the  time.  Further,  suppose  it  is  desired  to  select  a 
subset  which  includes  all  such  services  with  a  probability 
of  correct  selection  at  least  0.90.  Then  k  =  4,  Cl(thld) 
8.0,  C2(thld)  =  10.0,  pl(min)  =  0.40,  p2(min)  =  0.70,  and  P* 
=  0.90. 

Step  2:     Choose  n  =  90.  Ninety  independent 

measurements  have  been  made  on  each  computer  service. 


Step  3:     For     the  four 
indicates  that: 

Xl(l)=38 
Xl(2)=90 
Xl(3)=  0 
Xl(4)=78 


computer    services.     Table  4 

X2(l)=52  " 
X2  (2)=90 
X2(3)=  0 
X2  (4)=80 


Step  j4:  Table  8  in  Appendix  A  shows  that 
(M1,M2)  =  (27,52)  for  this  example.  Therefore  include 
services  1,  2  and  4  and  exclude  service  3,  i.e.  S(dual)  = 
{service(l),  service(2),  service(4)}.  Since  the  criterion 
was  desirable,  services  1,  2  and  4  would  be  appropriately 
rewarded. 


4.2.3  Selection    of    Services    Satisfying      an  Upper-Lower 
Proportion  Criterion  (1  example) 

Example 
(parallels  Section  3.2.3) 

Step  1:  Suppose  it  is  a  mandatory  criterion  that  a 
computer  service  be  able  to  perform  the  entire  script  (  1-10 
in  Table  3)  in  less  than  8.0  minutes  at  least  80%  of  the 
time.  Further,  suppose  it  is  desired  to  select,  with 
probability  of  correct  selection  0.90,  a  subset  which 
includes  all  such  services  and  excludes  any  service  which 
performs  the  script  in  less  than  8.0  minutes  less  than  60% 
of  the  time.  Then  k  =  4,  C(thld)=  8.0,  p_upper(min)  =  0.80, 
p_lower(max)  =  0.60,  and  P*  =  0.90. 
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step  2:  Choose  n  =  77  (based  on  Table  28  in  Appendix 
B) .  Ninety  independent  measurements  have  been  made  on  each 
computer  service;  ^  choose  the  first  77  measurements. 

Step  3:  For  the  four  computer  services.  Table  4 
indicates  that  X(l)=77,  X(2)=76,  X(3)=9  and  X(4)=43. 

Step  £:  Table  28  in  Appendix  B  shows  M  =  55  for  this 
example.  Therefore  include  services  1  and  2  and  exclude 
services  3  and  4,  i.e.  S (upper-lower )  =  {service(l), 
service(2)}.  since  the  criterion  was  mandatory,  services  3 
and  4  would  be  eliminated  from  the  selection  process. 


5.0  DISCUSSION  AND  CONCLUSIONS 

One  requirement  for  effective  computer  service 
comparison  in  a  selection  methodology  is  a  set  of  procedures 
which  can  provide  accurate  appraisals  of  service  behavior. 
Because  services  are  compared  in  an  uncontrolled 
environment,  the  data  collection  and  analysis  component  of 
the  selection  methodology  must  account  for  the  variability 
in  the  collected  measurements.  This  publication  indicates  a 
general  model  of  the  computer  service  selection  process 
based  on  the  application  of  four  classes  of  performance 
criteria.  For  use  in  phase  II  or  phase  IIIB  of  that  model, 
where  measurable  criteria  are  applied,  three  binary  type 
selection  procedures  are  given.  Based  on  statistical 
ranking  and  selection  theory,  assuming  one  can  collect 
statistically  independent  measurements,  the  procedures  have 
valid  application  when  measurable  selection  criteria  define 
computer  service  performance  in  terms  of  a  threshold 
level (s)   and  a  proportion  statistic (s) . 

Each  procedure  described  provides  information  regarding 
the  performance  of  a  set  of  computer  services  for  a 
particular  performance  index.  To  arrive  at  a  final 
selection  of  the  best  overall  service,  it  may  be  necessary 
to  integrate  the  information  obtained  from  individually 
applying  these  procedures  for  several  performance  indices 
with  cost  estimates  and  other  analysis  results.  Selection 
methodologies  for  integrating  all  of  the  available 
information  have  been  referenced. 

The  procedures  explicitly  specify  prior  to  measurement 
an  appropriate  decision  rule  and/or  the  number  of 
independent  measurements  required  from  each  computer  service 
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to  attain  a  given  level  of  statistical  confidence  in  the 
comparison  results.  The  procedures  are  valuable  in  the 
computer  service  procurement  process  because  they  design 
comparison  studies  to  result  in  statistical  confidence 
statements  about  the  probability  of  having  accurately 
estimated  computer  service  performance.  This  is  in  contrast 
to  those  studies  which  make  statements  about  which  computer 
services  satisfy  certain  performance  specifications,  but 
omit  statistical  justification  in  the  form  of  probabilistic 
confidence  statements. 

If  the  collection  of  measurements  is  necessary  in  a 
computer  service  selection,  then  these  procedures  (or 
similar  statistically  sound  procedures  such  as  those  in 
SP  500-44)  should  be  employed.  Their  mathematical 
derivation  provides  an  analyst  with  the  ability  to  justify 
the  empirical  results  employed  in  the  computer  service 
selection  decision  making  process. 
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APPENDIX  A 
Ta"bles  for  Dual  Proportion  Criteria 

This  appendix  contains  tabled  values  which  are  likely  to  he  needed 
to  apply  the  selection  procedure  described  in  Section  3.2.2.     A  demon- 
stration of  the  use  of  these  tables  can  be  found  in  Section  h.2.2. 
The  original  source  of  the  tables  is  [AMET9].     In  all  of  these  tables, 
"population"  represents  a  computer  service,  "dual  standard"  represents 
a  dual  proportion  criterion,  "P(CS)"  represents  the  probability  of 
correct  selection,  and  "standard  proportions  1  and  2"  represents  pl(min) 
and  p2(min),  respectively. 
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APPENDIX  B 

Tables  for  Upper-Lower  Proportion  Criteria 

This  appendix  contains  tabled  values  wRich  are  likely  to  be 
needed  to  apply  the  selection  procedure  described  in  Section  3.2.3. 
A  demonstration  of  the  use  of  these  tables  can  be  found  in  Section 
k.2.3.     The  original  source  of  the  tables  is  [AMET9].     In  all  of 
these  tables,  "population"  represents  a  computer  service,  "P(CS)" 
represents  the  probability  of  correct  selection,  and  "standard 
proportions  upper  and  lower"  represents  p_upper(min)  and  p_lo'wer (max) 
respectively. 
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Table  25 


VALUES  OF  n,M  FOR  SELECTING,   VITH  P(CS)   >=  P*,   A  SUBSET  OF  k  =  1 
BIHOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 


UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 


(POPULATIONS 

WITH 

NO. 

OF  'SUCCESSES' 

>=  M 

OUT 

OF  n 

TRIALS 

ARE 

SELECTED) 

P^jc  = 

1 

P*  = 

.95 

I 
1 

STANDARD  1 

1 

1 

PROPORTIONS! 

1 

1 

UPPER 

LOWER 1 

n 

M 

n 

M 

n 

M  1 

n 

M 

n 

M 

n 

ft  1 

.90 

j 

.80  1 

86 

1 

741 

172 

1 

1501 

258 

1 

226  1 

135 

1 

116  1 

270 

1 

235  1 

405 

1 

354  1 

.70  1 

25 

211 

50 

42  1 

75 

64  1 

41 

341 

82 

69  1 

123 

105  1 

.  60  1 

1  "tI 
1  O 

1  O  1 

30 

25  1 

45 

OO  1 

1  Q  1 

4ift 

40  1 

72 

60  1 

.  50  1 

9 

7  i 

18 

15  1 

27 

221 

13 

101 

26 

21  1 

39 

32  1 

.  40  1 

5 

41 

10 

8  1 

15 

121 

8 

6  1 

16 

121 

24 

19  1 

.30  1 

4 

31 

8 

6  1 

12 

9  1 

7 

5  1 

14 

1 1  1 

21 

16  1 

.20  1 

4 

31 

8 

6  1 

12 

9  1 

6 

4  1 

12 

9  1 

18 

141 

10  1 

1 

1  1 

3 

2  1 

3 

2  1 

3 

2  1 
1 

6 

41 
1 

9 

6  1 

1 

.80 

.70  1 

127 

96  1 

254 

1 

195  1 

381 

1 

295  1 

204 

1541 

408 

313  1 

6  12 

473  1 

.60  1 

36 

26  1 

72 

53  1 

108 

81  1 

60 

43  1 

120 

89  1 

180 

135  1 

.  50  1 

19 

13  1 

38 

27  1 

57 

42  1 

28 

19  1 

56 

40  1 

84 

61  1 

'.40  1 

9 

6  1 

18 

12  1 

27 

19  1 

17 

1 1  1 

34 

23  1 

5  1 

36  1 

.30  1 

6 

A  1 

4  1 

12 

8  1 

18 

1  u 

^  1 
O  1 

1  1 

30 

20  1 

.20  1 

5 

3  1 

10 

6  1 

15 

10  1 

7 

41 

14 

9  1 

21 

14  1 

,  10  1 

4 

21 

8 

5  1 

12 

81 

6 

31 
1 

12 

71 

1 

18 

1 1  1 

1 

.70 

.60  1 

loo 

1 

1  AO  1 

312 

1 

208  1 

468 

1 

Q  1  "tI  1 

O  1  O  1 

OO  V  1 

744 

500  1 

.50  1 

39 

241 

78 

49  1 

1 17 

76  1 

67 

41  1 

134 

85  1 

201 

130  1 

.40  1 

19 

11  1 

38 

23  1 

57 

35  1 

28 

16  1 

56 

33  1 

84 

521 

.30  1 

y 

O  1 

1  ft 

1  Ct  1 

27 

1  7 

y  1 

34 

19  1 

5  1 

30  1 

.20  1 

6 

3  1 

12 

6  1 

18 

10 1 

10 

5  1 

20 

1 1  1 

30 

17  1 

.  10  1 

A. 

^  1 

8 

41 

12 

f,  1 

1 

7 

3  1 

14 

7  1 

21 

1 1  1 

1 

.60 

.50  1 

168 

1 

931 

336 

1 

190  1 

504 

2881 

268 

1481 

536 

303  1 

804 

459  1 

.40  1 

41 

211 

82 

43  1 

123 

67  1 

67 

34  1 

134 

71  1 

201 

109  1 

.30  1 

19 

9  1 

38 

19  1 

57 

29  1 

28 

13  1 

56 

281 

84 

43  1 

.20  1 

9 

41 

18 

81 

27 

13  1 

17 

71 

34 

16  1 

51 

25  1 

.  10  1 

5 

21 

10 

41 

15 

71 

1 

8 

31 
1 

16 

6  1 
1 

24 

10  1 

1 

.50 

.40  1 

168 

1 

76  1 

336 

1 

156  1 

504 

2381 

268 

121  1 

536 

249  1 

804 

379  1 

.30  1 

39 

16  1 

78 

33  1 

1 17 

52  1 

67 

27  1 

134 

57  1 

201 

89  1 

.20  1 

19 

7  1 

38 

15  1 

57 

24  1 

28 

101 

56 

22  1 

84 

341 

.  10  1 

9 

31 

18 

6  1 

27 

101 

13 

41 
1 

26 

9  1 
1 

39 

14  1 

1 

.40 

.30  1 

156 

1 

55  1 

312 

1 

1141 

468 

1 

174  1 

248 

871 

496 

181  1 

744 

276  1 

.20  1 

36 

1 1  1 

72 

231 

108 

371 

60 

181 

120 

39  1 

180 

61  1 

.  10  1 

15 

41 

30 

9  1 

45 

14  1 

24 

6  1 
1 

48 

14  1 
1 

72 

22  1 
1 

.30 

.20  1 

127 

1 

32  1 

254 

1 

671 

381 

1 

1031 

204 

51  1 

408 

107  1 

612 

165  1 

.  10  1 

25 

5  1 

50 

1 1  1 

75 

171 

41 

81 

1 

82 

181 

1 

123 

29  1 

1 

.20 

.  10  1 

86 

1 

131 

172 

1 

281 

258 

1 

431 

135 

201 

270 

43  1 

405 

681 
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Table  26 


VALUES  OF  n.M  FOR  SELECTING,   WITH  P(CS)   >=  A  SUBSET  OF  k  =  2 

BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF   'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


i  P*  =   .90  I  P*  =   .95  I 


STANDARD  1 
PROPORTIONS! 
UPPER  LOWER 1 

n 

M 

n 

M 

n 

1 
1 

M  1 

n 

N 

n 

M 

n 

1 
1 

M  1 

1 

.90       . 80  1 

135 

1 

116  1 

270 

1 

235  1 

405 

1 

354  1 

190 

1 

163  1 

380 

1 

330  1 

570 

1 

499  1 

.70  1 

41 

34  1 

82 

69  1 

123 

105  1 

57 

471 

1 14 

96  1 

171 

146  1 

.60  1 

20 

16  1 

40 

331 

60 

50  1 

29 

231 

58 

47  1 

87 

72  1 

.50  1 

13 

10  1 

26 

21  1 

39 

32  1 

17 

131 

34 

27  1 

51 

41  1 

.40  1 

8 

6  1 

16 

12  1 

24 

19  1 

14 

101 

28 

221 

42 

34  1 

.30  1 

7 

5  1 

14 

11  1 

21 

16  1 

9 

6  1 

18 

131 

27 

21  1 

.  20  1 

6 

4  1 

12 

9  1 

18 

14  1 

6 

4  1 

12 

8  1 

18 

13  1 

.  10  1 

3 

21 

6 

4  1 

9 

6  1 

5 

3  1 

10 

7  1 

1 

15 

1 1  1 

.80       . 70  1 

200 

1 

151  1 

460 

1 

307  1 

600 

1 

464  1 

285 

1 

215  1 

570 

437  i 

855 

1 

661  1 

.60  1 

57 

41  1 

1 14 

84  1 

171 

1281 

80 

57  1 

160 

1 181 

240 

180  1 

.  50  1 

28 

19  1 

56 

40  1 

84 

6  1  1 

37 

25  1 

74 

52  1 

1 1 1 

80  1 

.  40  1 

17 

1 1  1 

34 

23  1 

5  1 

36  1 

22 

14  1 

44 

30  1 

66 

46  1 

.30  i 

10 

6  1 

20 

13  1 

30 

20  1 

15 

9  1 

30 

19  1 

45 

31  1 

.20  1 

7 

41 

14 

9  1 

21 

14  1 

9 

5  1 

18 

11  1 

27 

17  1 

.  10  1 

6 

3  1 
1 

12 

7  1 

18 

121 

6 

31 

12 

7  1 

18 

1 1  1 

.70       .60  1 

245 

1601 

490 

1 

326  1 

735 

1 

4941 

351 

1 

229  1 

675 

1 

449  1 

1000 

1 

671  1 

.50  1 

67 

41  1 

134 

85  1 

201 

130  1 

92 

56  1 

184 

1 16  1 

276 

1781 

.40  1 

28 

16  1 

56 

341 

84 

52  1 

44 

25  1 

88 

531 

132 

621 

•  t 

1  ^ 

1  o 

O  1 

1  y  1 
i  i  1 

OA  1 
1 

^  1  1 

.20  1 

10 

5  1 

20 

1 1 1 

30 

171 

15 

71 

30 

16  1 

45 

25  1 

.  10  1 

7 

3  1 

14 

7  1 

21 

1 1  1 

9 

41 

18 

9  1 

27 

14  1 

.60       .50  1 

266 

1 

147  1 

532 

1 

301  1 

798 

1 

456  1 

377 

1 

2081 

688 

1 

3881 

1000 

1 

570  1 

.40  1 

65 

331 

130 

69  1 

195 

106  1 

95 

481 

190 

101  1 

285 

155  1 

.30  1 

28 

13  1 

56 

281 

84 

43  1 

44 

20  1 

88 

44  1 

132 

681 

.20  1 

17 

71 

34 

16  1 

51 

25  1 

22 

9  1 

44 

20  1 

66 

321 

.  10  1 

8 

31 

16 

6  1 
1 

24 

10  1 

1 

14 

5  1 

28 

121 

42 

19  1 

.50       .40  1 

266 

1 

1201 

532 

2471 

798 

376  1 

377 

1 

1701 

688 

1 

3181 

1000 

1 

469  1 

.30  1 

67 

271 

134 

581 

201 

89  1 

92 

371 

184 

79  1 

276 

1221 

.20  1 

28 

10  1 

56 

22  1 

84 

35  1 

37 

131 

74 

29  1 

1 1 1 

45  1 

.  10  1 

13 

41 

26 

9  1 
1 

39 

14  1 

17 

5  1 

34 

1 1  1 

51 

19  1 

.40       .30  1 

245 

1 

86  1 

490 

1781 

735 

1 

272  1 

351 

1 

1231 

675 

1 

245  1 

1000 

1 

3701 

.20  1 

57 

17  1 

114 

37  1 

171 

581 

80 

24  1 

160 

52  1 

240 

81  1 

.  10  1 

20 

5  1 

40 

1 1  1 

60 

181 

29 

7  1 

58 

16  1 

87 

26  1 

.30       .20  1 

200 

1 

501 

400 

1 

105  1 

600 

1 

1621 

285 

1 

711 

570 

1 

1501 

855 

1 

231  1 

.  10  1 

41 

81 

82 

181 

123 

29  1 

57 

1 1  1 

1 14 

25  1 

171 

40  1 

.20       . 10  1 

135 

1 

20  1 

270 

1 

431 

405 

1 

681 
 1- 

190 

1 

281 

380 

1 

61  1 

570 

1 

96  1 
 1 

66 


Tatle  27 


VALUES  OF  n,M  FOR  SELECTING.   WITH  P(CS)   >=  A  SUBSET  OF  k  =  3 

BINOMIAL  POPULATIONS  WICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDAHD 
(POPULATIONS  WITH  NO.   OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


I                      P*  =    .90                     I                      P>tc  =  .95  I 

STANDARD     I                                                          I  I 

PROPORTIONS!                                                          I  I 

UPPER  LOWERI  n         M        n         M         n         MIn  M        n  M         n  Ml 


.90  .80 
.70 
.60 
.50 
.4© 
.30 
.20 
.  10 

.80  .70 
.60 
.50 
.40 
.30 
.20 
.  10 

.70  .60 
.50 
.40 
.30 
.20 
.  10 

.60  .50 
,40 
.30 
.20 
.  10 

.50  .40 
.30 
.20 
.  10 

.40  .30 
.20 
.  10 

.30  .20 
.  10 

.20  .10 


169 

145  1 

4C>  1 
1 

20  1 

1  7 

1  3  1 

1  1 

81 

7 

5  1 

(f 

4  1 

\j 

249 

1 

1881 

tj  V  1 

34 

23  1 

20 

131 

13 

8  1 

7 

4  1 

6 

31 

305 

1 

199  1 

82 

50  1 

37 

21  1 

19 

10  1 

13 

6  1 

7 

31 

328 

1 

181  1 

81 

41  1 

37 

171 

20 

81 

1 1 

4  1 

328 

1 

1481 

82 

33  1 

34 

12  1 

17 

5  1 

305 

i 

107  1 

70 

21  1 

25 

6  1 

249 

1 

62  1 

51 

10  1 

169 

1 

25  1 

I 


338 

2941 

1  fti> 

Rfi  1 
o\}  1 

4<  1  1 

27  1 

22 

17  1 

14 

10  1 

1  2 

0  1 

V  1 

4i  1 

498 

1 

3821 

1  d.ft 

1  ft'^  1 

68 

481 

40 

271 

26 

17  1 

14 

81 

12 

7  1 

610 

1 

406  1 

164 

1041 

74 

45  1 

38 

21  1 

26 

14  1 

14 

71 

656 

1 

371  1 

162 

86  1 

74 

371 

40 

181 

22 

9  1 
1 

656 

305  1 

164 

701 

68 

271 

34 

12  1 

610 

1 

2221 

140 

46  1 

50 

14  1 

498 

1 

131  1 

102 

221 

338 

1 

541 

507 

444  1 

1 

1       1  1 

io  1  1 

€  %J 

TJo  1 

O  1 

1 

tt^  1 

33 

26  1 

21 

16  1 

1  o 

1  4i  1 

I  1 

Q 

A  1 
O  1 

747 

1 

5781 

1  17  1 

102 

74  1 

60 

421 

2fi  1 

21 

131 

18 

111 

915 

1 

615  1 

246 

159  1 

1 1 1 

69  1 

57 

331 

39 

22  1 

21 

1 1  1 

984 

1 

5621 

243 

1321 

1 1 1 

571 

60 

29  1 

33 

15  1 
1 

984 

4631 

246 

109  1 

102 

42  1 

51 

19  1 

! 

915 

339  1 

210 

71  1 

75 

22  1 
1 

747 

201  1 

153 

36  1 
1 

507 

85  1 

I 


225 

1931 

t)0 

Do  1 

^  t  t 

O  1 

ID  1 

15 

1 1 1 

10 

7  1 

Q 

o 

O  1 

o 

Q  1 
o  1 

337 

1 

2541 

Q  A. 

O  <  1 

43 

29  1 

27 

17  1 

1  1 

1  1 

1 1 

6  1 

8 

4  1 

414 

1 

270  1 

107 

65  1 

48 

27  1 

27 

14  1 

17 

81 

10 

4  1 

446 

1 

246  1 

1 1 1 

56  1 

48 

221 

27 

1 1  1 

15 

5  1 
1 

446 

201  1 

107 

431 

43 

15  1 

21 

6  1 

414 

1 

145  1 

94 

281 

34 

81 

1 

337 

841 

68 

13! 

1 

225 

33  1 

I 


450 

39  11 

1  oo 

1  lo  1 

OO 

DO  1 

QQ  1 

30 

231 

20 

15  1 

1  o 

1  1  1 

1  k) 

7  1 

668 

1 

512  1 

loo 

loo  1 

86 

61  1 

54 

371 

22 

131 

16 

9  1 

707 

1 

469  1 

214 

135  1 

96 

571 

54 

301 

34 

181 

20 

9  1 

723 

1 

406  1 

222 

1 181 

96 

471 

54 

25  1 

30 

121 

723 

1 

3331 

214 

921 

86 

33  1 

42 

14  1 

707 

1 

255  1 

188 

611 

68 

19  1 

668 

1 

176  1 

136 

30  1 

450 

1 

721 

675 

591  1 

IT"*  1 

1  /AO 

Qc;  1 
OO  \ 

OO 

O  1  1 

45 

36  i 

30 

23  1 

lo  1 

1  Pt 

lo 

1  1  1 
1  1  1 

1000 

1 

7731 

O  1  1  1 

^111 

129 

931 

81 

571 

O  1 

QA  1 

33 

211 

24 

15  1 

1000 

1 

669  1 

321 

2071 

144 

89  1 

81 

481 

51 

29  1 

30 

16  1 

1 

1000 

5671 

333 

181  1 

144 

741 

81 

39  1 

45 

201 

1000 

1 

466  1 

321 

1421 

129 

521 

63 

231 

1000 

I 

367  1 

282 

95  1 

102 

30  1 

1000 

1 

2701 

204 

481 

675 

1 

1131 

67 


Table  28 


VALUES  OF  n,M  FOR  SELECTING,   WITH  P(CS)   >=  P*,   A  SUBSET  OF  k  =  4 
BINOMIAL  POPULATIONS  raiCH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF   'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


P*  = 

.90 

1 

P*  = 

.95 

1 

STANDARD  1 

1 

1 

PROPORTIONS! 

1 

1 

UPPER 

LOWER 1 

n 

M 

n 

M 

n 

M  1 

n 

M 

n 

M 

n 

K  1 

.  90 

.  80  1 

190 

1 

163  1 

380 

1 

330  1 

570 

1 

499  1 

252 

1 

216  1 

504 

1 

4381 

756 

1 

66  1  1 

.  70  1 

57 

47  1 

1  14 

96  1 

171 

146  1 

78 

641 

156 

1321 

234 

200  1 

.  60  1 

29 

23  1 

58 

47  1 

87 

73  1 

38 

301 

76 

621 

1 14 

95  1 

.  50  1 

17 

13  1 

34 

27  1 

5  1 

4  1  1 

24 

181 

48 

381 

72 

59  1 

!40  1 

12 

9  1 

24 

181 

36 

29  1 

15 

1 1  1 

30 

23  1 

45 

36  1 

.30  1 

9 

6  1 

18 

13  1 

27 

21  1 

12 

8  1 

24 

181 

36 

28  1 

.  20  1 

6 

4  1 

12 

9  1 

18 

13  1 

8 

5  1 

16 

1 1  1 

24 

18  1 

.  10  1 

5 

3  1 

10 

7  1 

15 

1 1  1 

5 

3  1 

10 

6  1 

15 

1 1  1 

.80 

.70  1 

281 

1 

212  1 

562 

1 

431  1 

843 

1 

652  1 

374 

1 

282  1 

687 

526 1 1000 

1 

771  1 

.60  1 

77 

55  1 

154 

1  13  1 

23  1 

173  1 

104 

74  1 

208 

153  1 

312 

233  1 

.  50  1 

37 

25  1 

74 

52  1 

1 1 1 

80  1 

49 

33  1 

98 

69  1 

147 

106  1 

.  40  1 

22 

14  1 

44 

30  1 

66 

46  1 

30 

19  1 

60 

41  1 

90 

63  1 

^30  1 

15 

9  1 

30 

20  1 

45 

31  1 

17 

10  1 

34 

22  1 

51 

34  1 

.20  1 

9 

5  1 

18 

1  1  1 

27 

17  1 

1 1 

6  1 

22 

13  1 

33 

21  1 

10  1 

6 

3  1 
1 

12 

7  1 
1 

18 

1 1  1 

1 

8 

4  1 

16 

9  1 

24 

15  1 

.70 

.60  1 

348 

227  1 

674 

448  1 

1000 

672  1 

457 

1 

298  1 

728 

482 1 1000 

1 

667  1 

.50  1 

92 

56  1 

184 

1 17  1 

276 

1781 

120 

73  1 

240 

1521 

360 

232  1 

OiCi  1 
■  *T\r  1 

44 

25  1 

88 

53  1 

132 

82  1 

55 

31  1 

1 10 

66  1 

165 

1021 

.30  1 

23 

12  1 

46 

26  1 

69 

41  1 

29 

15  1 

58 

33  1 

87 

5  1  1 

.20  1 

15 

7  1 

30 

16  1 

45 

25  1 

17 

81 

34 

181 

5  1 

281 

.  10  1 

9 

4  1 

18 

9  1 

27 

14  1 

12 

5  1 

24 

121 

36 

19  1 

.60 

.50  1 

377 

1 

2081 

688 

1 

3881 

1000 

1 

5701 

495 

1 

2731 

747 

1 

4181 

1000 

1 

565  1 

.40  1 

93 

47  1 

186 

99  1 

279 

151  1 

123 

62  1 

246 

130  1 

369 

200  1 

.30  1 

44 

20  1 

88 

44  1 

132 

681 

55 

25  1 

1 10 

54  1 

165 

85  1 

.20  1 

22 

9  1 

44 

20  1 

66 

32  1 

30 

12  1 

60 

27  1 

90 

44  1 

.  10  1 

12 

41 

24 

101 

36 

16  1 

15 

5  1 

30 

12  1 

[ 

45 

20  1 

.50 

.40  1 

377 

1 

1701 

688 

1 

319  1 

1000 

1 

469  1 

495 

1 

2231 

747 

343 1 1000 

1 

465  1 

.30  1 

92 

37  1 

184 

79  1 

276 

122  1 

120 

481 

240 

103  1 

360 

159  1 

.20  1 

37 

13  1 

74 

29  1 

1 1 1 

45  1 

49 

17  1 

98 

381 

147 

60  1 

.  10  1 

17 

5  1 

34 

1 1  1 

51 

19  1 

24 

7  1 

48 

16  1 

72 

27  1 
1 

.40 

.30  1 

343 

1 

122  1 

674 

1 

245  1 

1000 

1 

370  1 

457 

1 

1601 

728 

1 

2621 

1000 

366  1 

.20  1 

77 

23  1 

154 

501 

231 

781 

104 

31  1 

208 

681 

312 

106  1 

.  10  1 

29 

7  1 

58 

16  1 

87 

26  1 

38 

9  1 

1 

76 

21  1 

1 

1 14 

34  1 

1 

.30 

.20  1 

281 

1 

70  1 

562 

1 

1481 

843 

1 

227  1 

374 

93  1 

687 

180 1 1000 

2681 

.  10  1 

57 

1 1  1 

1  14 

25  1 

171 

40  1 

78 

15  1 

156 

341 

234 

55  1 

1 

.20 

.  10  1 

190 

1 

281 

380 

1 

611 

570 

1 

96  1 
 1- 

252 

1 

371 

504 

1 

81  1 

756 

127  1 
 1 

68 


Table  29 


VALUES  OF  n,M  FOR  SELECTING,   WITH  P(CS)   >=  ?*,   A  SUBSET  OF  k  =  5 
BINOMIAL  POPULATIONS  WICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.    OF   'SUCCESSES'    >=   M  OUT  OF  n  TRIALS  ARE  SELECTED) 


1  P*  =    .90  )  P*  =  .95 

STANDARD  I  I 
PROPORTIONS!  I 

UPPER  LOt^ERI  nMnMnMlnPInNnM 


I 


.  90 

.  80  1 

210 

180  1 

420 

365  1 

630 

551  1 

266 

228  1 

532 

462  1 

798 

6981 

.70  ! 

62 

511 

124 

104  1 

186 

159  1 

83 

681 

166 

140  1 

249 

213  1 

.  60  1 

33 

26  1 

66 

54  1 

99 

83  1 

42 

33  1 

84 

69  1 

126 

105  1 

.  50  1 

20 

15  1 

40 

32  1 

60 

49  1 

25 

19  1 

50 

40  1 

75 

6  1  1 

.40  1 

14 

10  1 

28 

22  1 

42 

33  1 

18 

13  1 

36 

28 1 

54 

43  1 

.  30  1 

10 

7  1 

20 

15  1 

30 

23  1 

12 

81 

24 

181 

36 

281 

.20  1 

6 

4  1 

12 

81 

18 

13  1 

9 

6  1 

18 

131 

27 

20  1 

.  10  1 

5 

3  1 

10 

7  1 

15 

1 1  1 

5 

3  1 
1 

10 

6  1 

1 

15 

10  1 

1 

.  80 

1 

.  70  1 

309 

1 

233  1 

618 

1 

474  1 

927 

1 

716  1 

402 

303  1 

701 

536  1 

1000 

770  1 

.60  1 

87 

62  1 

174 

1281 

261 

195  1 

1 1 1 

79  1 

222 

163  1 

333 

249  1 

.50  1 

40 

27  1 

80 

56  1 

120 

87  1 

52 

35  1 

104 

73  1 

156 

113  1 

.  40  1 

25 

16  1 

50 

34  1 

75 

53  1 

30 

19  1 

60 

40  1 

90 

63  1 

.30  1 

15 

9  1 

30 

19  1 

45 

301 

20 

12  1 

40 

26  1 

60 

40  1 

.  20  1 

9 

5  1 

18 

1 1  1 

27 

17  1 

lo 

(  1 

1  ^  1 

ID  1 

t 

.  10  1 

6 

3  1 

12 

71 

13 

11  1 

9 

41 

1 

18 

101 

1 

27 

16  1 

1 

.70 

1 

.60  1 

377 

1 

246  1 

688 

1 

457  1 

1000 

1 

6701 

486 

3171 

743 

491  1 

1000 

666  1 

.50  1 

102 

62  1 

204 

129  1 

306 

1981 

130 

79  1 

260 

165  1 

390 

252  1 

.40  1 

46 

26  1 

92 

55  1 

138 

85  1 

57 

32  1 

1 14 

681 

171 

106  1 

.30  1 

1  25 

13  1 

50 

281 

75 

44  1 

31 

16  1 

62 

35  1 

93 

55  1 

.20  1 

1  15 

71 

30 

16  1 

45 

25  1 

20 

9  1 

40 

21  1 

60 

34  1 

.  10  1 

1  10 

41 

20 

10  1 

30 

16  1 

12 

5  1 
1 

24 

11  1 

1 

36 

19  1 
1 

.60 

1 

.50  1 

I 

i  415 

1 

229  1 

707 

1 

3981 

1000 

1 

5681 

535 

295  1 

767 

429  1 

1000 

564  1 

.40  1 

1  103 

52  1 

206 

109  1 

309 

1681 

133 

67  1 

266 

141  1 

399 

217  1 

.30  I 

1  46 

21  1 

92 

46  1 

138 

71  1 

57 

26  1 

1 14 

56  1 

171 

831 

.20 

1  25 

10  1 

50 

23  1 

75 

36  1 

30 

121 

60 

27  1 

90 

43  1 

.  10 

1  14 

5  1 

28 

1 1  1 

42 

19  1 

1 

18 

6  1 

1 

36 

15  1 

1 

54 

24  1 

1 

,50 

.40 

1 

1  415 

1 

187  1 

707 

1 

326  1 

1000 

4681 

535 

241  1 

767 

351  1 

1000 

463  1 

.30 

1  102 

41  1 

204 

87  1 

306 

135  1 

130 

521 

260 

nil 

390 

172  1 

.20 

1  40 

14  1 

80 

31  1 

120 

49  1 

52 

181 

104 

40  1 

156 

64  1 

.  10 

1  20 

6  1 

40 

14  1 

60 

221 

1 

25 

7  1 

1 

50 

17  1 

1 

75 

231 

1 

.40 

.30 

1 

1  377 

1 

132  1 

688 

1 

249  I 

1000 

369  1 

486 

1701 

743 

266  1 

1000 

3641 

.20 

1  87 

26  1 

174 

57  1 

261 

881 

1 1 1 

331 

222 

72  1 

333 

1 13  1 

.  10 

1  33 

81 

66 

181 

99 

301 

42 

10  1 
1 

84 

23  1 

1 

126 

381 

1 

.30 

.20 

1 

1  309 

1 

771 

618 

1 

1621 

927 

1 

2501 

402 

100  1 

701 

182  1 

1000 

267  1 

.  10 

1  62 

12  1 

124 

27  1 

186 

43  1 
1 

83 

16  1 

1 

166 

36  1 
1 

249 

581 

1 

.20 

.  10 

1 

1  210 

1 

31  1 

420 

1 

681 

630 

106  1 

266 

39  1 

532 

85  1 

798 

1341 

69 


Table  30 


VALUES  or  n,M  FOR  SELECTING,   WITH  PCCS)   >=  P*,   A  SUBSET  OF  k  =  6 
BINOMIAL.  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDAiU)  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


r   ■  - 

1 
1 

P*  = 

.  yo 

1 

STANDARD  1 

1 

1 

PROPORTIONS! 

1 

! 

UPPER 

LOIVERI 

11 

M 

n 

M 

n 

M  1 

n 

M 

n 

M 

n 

M  ! 

.90 

1 

.80  1 

224 

1 

1921 

448 

1 

389  1 

672 

! 

588! 

286 

I 

245  1 

572 

! 

497! 

858 

! 

751  ! 

.70  1 

oo 

1 

Dt)  1 

1 

1  OtJ 

1   1  1 

1  i  D  1 

1  '7A  1 
1  <  ^  1 

O  Q 
OD 

1  *  o 

OA  ^ 

o  o  1 

.60  1 

34 

27  1 

68 

56  1 

102 

85  ! 

46 

36  ! 

92 

75  1 

138 

115  ! 

.50  1 

21 

16  1 

42 

33! 

63 

51  ! 

28 

21  ! 

56 

45  1 

84 

68! 

.  4W  1 

15 

1 1  1 

30 

231 

45 

36  ! 

18 

131 

36 

281 

54 

43! 

.30  1 

10 

71 

20 

15  ! 

30 

23  ! 

13 

9  ! 

26 

19  ! 

39 

30  ! 

.20  1 

6 

4  1 

12 

8! 

18 

13  ! 

9 

6  ! 

18 

13! 

27 

20! 

.  10  1 

Q  I 
o  1 

1  Ck 

1  VJ 

T  1 
1  1 

1  o 

1  1  1 
1  1  1 

! 

9  ! 

21 

lO  1 

'"7  f\  1 

.  70  1 

333 

1 

251  1 

666 

! 

511! 

999 

772  ! 

426 

! 

321  ! 

713 

! 

544  ! 

1000 

1 

769  1 

.60  1 

94 

671 

188 

139  ! 

282 

211  1 

1 18 

84! 

236 

174  ! 

354 

265  1 

.50  1 

43 

29  1 

86 

61  ! 

129 

93  1 

55 

37  ! 

1 10 

781 

165 

1 19  ! 

.40  1 

1  ^  r 
1  o  1 

ou 

Q-d.  1 

o"*  1 

<  O 

OA  1 

43  1 

96 

.  dO  1 

17 

10 1 

34 

22  ! 

51 

34  ! 

22 

13! 

44 

281 

66 

45  ! 

.20  1 

11 

6  1 

22 

13! 

33 

21  ! 

13 

7! 

26 

16  1 

39 

25  ! 

.  10  1 

6 

3  1 

12 

6  1 

18 

1  1  ! 

9 

4! 

18 

10  ! 

27 

16  ! 

.70 

.60  1 

409 

267  1 

704 

467  ! 

1000 

669  ! 

520 

339  ! 

760 

! 

5021 

1000 

665  I 

.  50  1 

107 

65  1 

214 

135  ! 

321 

207  ! 

135 

82! 

270 

171  ! 

405 

261  1 

.40  1 

OT  1 

Oo  1 

1  /H,  A, 

DQ  1 

Q  1 

oO  1 

1  O/l. 

74  1 

186 

1  lO  1 

.30  1 

27 

14  1 

54 

31  1 

81 

48! 

33 

17! 

66 

37  1 

99 

58! 

.20  1 

17 

81 

34 

181 

51 

29  ! 

22 

10  ! 

44 

231 

66 

37! 

.  10  1 

10 

4  1 

20 

10  ! 

30 

16  ! 

13 

5  ! 

26 

121 
1 

39 

20  1 

.60 

.50  1 

444 

1 

245  1 

722 

! 

405  ! 

1000 

1 

5671 

566 

1 

312! 

783 

437 ! 1000 

1 

5631 

.  40  1 

109 

55  1 

218 

1 15  ! 

327 

177  ! 

139 

70  ! 

278 

147  1 

417 

226  ! 

.30  1 

48 

22  1 

96 

47  ! 

144 

74! 

62 

28! 

124 

61  1 

186 

96  ! 

.20  1 

25 

10  1 

50 

23  1 

75 

36  ! 

32 

13  ! 

64 

29  1 

96 

46  ! 

.  10  1 

15 

5  1 

30 

12  1 

45 

20  1 

18 

6  1 

36 

15  ! 

1 

54 

24! 

.50 

.40  1 

444 

1 

200  1 

722 

1 

333! 

1000 

1 

467  1 

566 

! 

255  ! 

783 

358! 1000 

! 

462! 

.30  1 

107 

43  1 

214 

92  ! 

321 

142  1 

135 

54  ! 

270 

1 15  1 

405 

179  ! 

.20  1 

43 

15  1 

86 

33! 

129 

53  ! 

55 

19  ! 

1 10 

43  1 

165 

671 

.  10  1 

21 

6  1 

42 

14  ! 

63 

23  ! 

28 

81 

56 

19  1 
1 

84 

31  1 

.40 

.30  1 

409 

1 

143  1 

704 

! 

254  ! 

1000 

! 

367  ! 

520 

1 

1821 

760 

272! 1000 

! 

363! 

.20  1 

94 

281 

188 

61  ! 

282 

96  ! 

1 18 

35  1 

236 

77! 

354 

120! 

.  10  1 

34 

81 

68 

19  1 

102 

31  1 

46 

1 1  ! 

! 

92 

26  1 

138 

421 

1 

.30 

.20  1 

333 

1 

831 

666 

1 

175  ! 

999 

1 

269  ! 

426 

106  ! 

713 

1 

185  1 

1000 

266  ! 

.  10  1 

68 

13  1 

136 

30! 

204 

48! 

88 

171 

176 

39  ! 

264 

62! 
! 

.20 

.  10  1 

224 

1 

33  1 

448 

! 

72! 

672 

! 

1131 
 !- 

286 

! 

42  ! 

572 

! 

92  1 

858 

144  ! 

 1 

70 


Table  31 


VALUES  OF  n,M  FOR  SELECTING,   WITH  P(CS)   >=  F*,   A  SUBSET  OF  k  =  7 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF   'SUCCESSES'   >=  N  OUT  OF  n  TRIALS  ARE  SELECTED) 


P*  = 

.  90 

1 

P*  = 

.  95 

1 

STANDARD  1 

1 

1 

PROPORTIONS! 

1 

1 

UPPER 

LOWER 1 

n 

M 

n 

N 

n 

M  1 

n 

M 

n 

M 

n 

M  1 

.90 

1 

.80  1 

238 

1 

204  1 

476 

1 

414  1 

714 

1 

625  1 

300 

1 

2571 

600 

1 

521  1 

900 

1 

787  1 

.70  1 

73 

60  1 

146 

123  1 

219 

187  1 

89 

73  1 

178 

150  1 

267 

228  1 

.60  1 

37 

29  1 

74 

61  1 

1 1 1 

93  1 

46 

36  1 

92 

75  1 

138 

1 15  1 

.50  1 

21 

16  1 

42 

33  1 

63 

51  1 

28 

21  1 

56 

44  1 

84 

631 

ACt  1 

.  TJV  1 

15 

1 1  1 

30 

23  1 

45 

36  1 

18 

13  1 

36 

27  1 

54 

43  1 

.30  1 

10 

^  1 

20 

15  1 

30 

23  1 

13 

9  1 

26 

19  1 

39 

30  1 

.20  1 

8 

5  1 

16 

1 1  1 

24 

181 

10 

6  1 

20 

14  1 

30 

22  1 

.  10  1 

5 

3  1 

10 

7  1 

15 

1 1  1 

7 

4  1 

14 

9  1 

2 1 

15  1 

R0 

.  70  1 

354 

1 

267  1 

677 

1 

519  1 

1000 

1 

772  1 

446 

1 

336  1 

723 

1 

552  1 

1000 

1 

769  1 

!60  1 

100 

71  1 

200 

147  1 

300 

225  1 

125 

89  1 

250 

184  1 

375 

281  1 

.50  1 

46 

31  1 

92 

65  1 

138 

100  1 

58 

39  1 

1 16 

82  1 

174 

126  1 

.40  1 

27 

17  1 

54 

36  1 

81 

57  1 

35 

22  1 

70 

47  1 

105 

74  1 

.  30  1 

17 

10  1 

34 

221 

51 

34  1 

22 

131 

44 

281 

66 

441 

!20  1 

1 1 

6  1 

22 

13  1 

33 

21  1 

13 

7  1 

26 

15  1 

39 

25  1 

.  10  1 

8 

4  1 

16 

9  1 

24 

15  1 

1 

10 

5  1 
1 

20 

1 1  1 

1 

30 

181 

1 

.70 

.60  1 

434 

1 

2831 

717 

1 

475  1 

1000 

6681 

543 

354  1 

771 

5081 

1000 

664  1 

.50  1 

1 12 

681 

224 

142  1 

336 

217  1 

145 

88  1 

290 

184  1 

435 

281  1 

.40  1 

53 

30  1 

106 

64  1 

159 

99  1 

64 

36  1 

128 

77  1 

192 

1 19  1 

.30  1 

27 

141 

54 

301 

81 

481 

35 

181 

70 

39  1 

105 

62  1 

.20  1 

17 

81 

34 

181 

51 

281 

22 

101 

44 

23  1 

66 

37  1 

.  10  1 

10 

4  1 

20 

9  1 

30 

15  1 

1 

13 

5  1 
1 

26 

12  1 
1 

39 

20  1 
1 

.60 

.50  i 

466 

1 

2571 

733 

1 

411  1 

1000 

566  1 

595 

3281 

797 

444  1 

1000 

5621 

.40  1 

117 

59  1 

234 

1241 

351 

191  1 

147 

74  1 

294 

156  1 

441 

239  1 

.3©  1 

53 

24  1 

106 

53  1 

159 

821 

64 

29  1 

128 

63  1 

192 

981 

.20  1 

27 

1 1  1 

54 

25  1 

81 

39  1 

35 

141 

70 

32  1 

105 

51  1 

.  10  1 

15 

5  1 

30 

121 

45 

201 

18 

6  1 
1 

36 

14  1 

1 

54 

24  1 
1 

.50 

.40  1 

466 

1 

2101 

733 

1 

3371 

1000 

1 

466  1 

595 

2681 

797 

3641 

1000 

461  1 

.30  1 

112 

45  1 

224 

96  1 

336 

1481 

145 

581 

290 

124  1 

435 

192  1 

.20  1 

46 

16  1 

92 

36  1 

138 

56  1 

58 

201 

1 16 

45  1 

174 

71  1 

.  10  1 

21 

6  1 

42 

14  1 

63 

23  1 

28 

81 

1 

56 

19  1 

1 

84 

31  1 
1 

.40 

.30  1 

434 

1 

152  1 

717 

1 

2581 

1000 

1 

366  1 

543 

190  1 

771 

275  1 

1000 

362  1 

.20  1 

100 

301 

200 

65  1 

300 

1021 

125 

371 

250 

81  1 

375 

127  1 

.  10  1 

37 

9  1 

74 

21  1 

1 1 1 

33  1 

46 

1 1  1 

1 

92 

26  1 
1 

138 

41  1 
1 

.30 

.20  1 

354 

1 

881 

677 

1 

1771 

1000 

1 

269  1 

446 

1 1  1  1 

723 

187  1 

1000 

265  1 

.  10  1 

73 

141 

146 

321 

219 

511 

89 

171 

1 

178 

39  1 

1 

267 

62  1 
1 

.20 

.  10  1 

238 

1 

35  1 

476 

1 

771 

714 

1 

1201 

300 

44  1 

600 

971 

900 

151  1 

71 


Table  32 


VALUES  OF  n,M  FOR  SELECTING,   WITH  P(CS)   >=  P*,   A  SUBSET  OF  k  =  O 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAJf  A  LO^/ER  STANDARD 
(POPULATIONS  WITH  NO.   OF   'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


P*  = 

.90 

1 

P:ic  = 

.95 

1 

STANDARD  1 

1 

1 

PROPORTIONS! 

1 

1 

UPPER 

LOWER 1 

n 

M 

n 

M 

n 

M  1 

n 

M 

n 

M 

n 

K  1 

.90 

1 

.80  1 

245 

1 

2101 

490 

1 

426  1 

735 

1 

643  1 

307 

1 

2631 

614 

1 

533  1 

921 

1 

805  1 

.  70  1 

73 

60  1 

146 

123  1 

219 

187  1 

94 

77  1 

188 

1581 

282 

241  1 

.  60  1 

38 

30  1 

76 

621 

1 14 

95  1 

47 

371 

94 

771 

141 

1171 

.  50  1 

24 

18  1 

43 

38  1 

72 

59  1 

28 

2 1  1 

56 

44  1 

84 

68  1 

.  40  1 

15 

1 1  1 

30 

23  1 

45 

36  1 

21 

15  1 

42 

32  1 

63 

50  1 

.30  1 

10 

7  1 

20 

15  1 

30 

23  1 

15 

10  1 

30 

22  1 

45 

35  1 

.20  1 

8 

5  1 

16 

1  1  1 

24 

181 

10 

6  1 

20 

14  1 

30 

22  1 

.  10  1 

5 

3  1 

10 

7  1 

15 

1  1  1 

7 

4  1 

14 

9  1 

21 

15  1 

.  80 

.  70  1 

370 

1 

279  1 

685 

1 

524  1 

1000 

1 

771  1 

466 

1 

35  1  1 

733 

1 

559  1 

1000 

1 

7681 

!60  1 

101 

72  1 

202 

149  1 

303 

227  1 

128 

9  1  1 

256 

188  1 

384 

287  1 

.50  1 

49 

33  1 

98 

69  1 

147 

106  1 

6  1 

41  1 

122 

86  1 

183 

132  1 

.  40  1 

30 

19  1 

60 

4 1  1 

90 

63  1 

35 

22  1 

70 

47  1 

105 

73  1 

.  30  1 

17 

10  1 

34 

22  1 

5  1 

34  1 

22 

13  1 

44 

281 

66 

441 

!20  1 

1  1 

6  1 

22 

13  1 

33 

21  1 

15 

81 

30 

18  1 

45 

29  1 

.  10  1 

8 

4  1 

16 

9  1 

24 

15  1 

10 

5  1 

20 

1  1  1 

30 

181 

,70 

.60  1 

454 

1 

296  1 

727 

1 

481  1 

1000 

1 

6681 

566 

1 

369  1 

783 

1 

516  1 

1000 

1 

664  1 

.5©  1 

1 17 

71  1 

234 

1481 

35  1 

226  1 

150 

9  1  1 

300 

190  1 

450 

290  1 

.40  1 

55 

3  1  1 

110 

66  1 

163 

102  1 

66 

37  1 

132 

79  1 

198 

122  1 

•  1 

29 

15  1 

58 

331 

87 

51  1 

37 

19  1 

74 

421 

1 1 1 

65  1 

,20  1 

17 

81 

34 

181 

5  1 

281 

22 

10  1 

44 

23  1 

66 

37  1 

.  10  1 

10 

4  1 

20 

9  1 

30 

15  1 

15 

6  1 

30 

15  1 

45 

241 

.60 

,50  1 

495 

1 

273  1 

747 

1 

4131 

1000 

1 

565  1 

615 

1 

339  1 

807 

1 

449  1 

1000 

1 

561  1 

.40  1 

121 

61  1 

242 

1281 

363 

197  1 

153 

771 

306 

162  1 

459 

249  1 

.30  1 

55 

25  1 

1 10 

54  1 

165 

85  1 

66 

301 

132 

65  1 

198 

102  1 

.20  1 

30 

12  1 

60 

27  1 

90 

44  1 

35 

14  1 

70 

32  1 

105 

501 

.  10  1 

15 

5  1 

30 

121 

43 

20  1 

21 

71 

1 

42 

171 

1 

63 

281 
1 

.50 

.40  1 

495 

1 

223  1 

747 

1 

343  1 

1000 

1 

465  1 

615 

277  1 

807 

3681 

1000 

461  1 

.30  1 

1 17 

47  1 

234 

100  1 

351 

155  1 

150 

601 

300 

1281 

450 

199  1 

.20  1 

49 

17  1 

98 

381 

147 

60  1 

61 

21  1 

122 

471 

183 

75  1 

.  10  1 

24 

7  1 

48 

16  1 

72 

27  1 

28 

81 

56 

19  1 

84 

311 

.40 

.30  1 

454 

1 

159  1 

727 

1 

262  1 

1000 

1 

366  1 

566 

1 

1981 

783 

1 

279  1 

1000 

1 

362  1 

.20  1 

101 

30  1 

202 

65  1 

303 

102  1 

128 

381 

256 

83  1 

384 

130  1 

.  10  1 

38 

9  1 

76 

21  1 

1 14 

34  1 

47 

1  1  1 

94 

26  1 

1 

141 

42  1 
1 

.30 

.20  1 

370 

1 

92  1 

685 

1 

179  1 

1000 

1 

2681 

466 

1 

1 16  1 

733 

189  1 

1000 

264  1 

.  10  1 

73 

14  1 

146 

32  1 

219 

51  1 
1 

94 

181 

1 

188 

41  1 

1 

282 

66  1 

1 

.20 

.  10  1 

245 

1 

36  1 

490 

1 

79  1 

735 

123  1 
 1- 

307 

45  1 

614 

99  1 

921 

155  1 
 1 

72 


Table  33 


VALUES  OF  n.M  FOR  SELECTING,   WITH  P(CS)   >=  ?*,   A  SUBSET  OF  k  =  9 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  TOAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


I  P*  =   .90                     I                      P*  =    .95  I 

STANDARD     I  I  I 

PROPORTIONS  I  I  I 

UPPER  LOWERI  n         M        n         N        n         MIn         M         n         M        n  Ml 


I  I  I  I  I 


.90 

.80  1 

259 

2221 

518 

450  1 

777 

6801 

321 

275  1 

642 

5581 

963 

8421 

TO 

{a 

1 

loo 

loll 

OAA  1 

1 

Q  1  1 
Ol  1 

lyti 

lO  (  1 

00*7 

<sy  < 

^04<  1 

.60  1 

38 

30  1 

76 

621 

1 14 

95  1 

51 

401 

102 

83  1 

153 

1281 

.50  1 

24 

181 

48 

381 

72 

59  1 

31 

231 

62 

49  1 

93 

76  1 

.40  1 

17 

121 

34 

26  1 

51 

41  1 

21 

15  1 

42 

32  1 

63 

501 

.30  1 

12 

81 

24 

181 

36 

281 

15 

101 

30 

221 

45 

35  1 

1  Q 
1  O 

O  1 

1  ^ 

lO 

1 1  1 

24 

1  O  1 

1 1 

"7  1 
(  1 

i  D  1 

OO 

1 
1 

.  10  1 

5 

3! 

10 

6  1 

15 

10  1 

7 

41 

14 

9  i 

1 

21 

15  1 

1 

.80 

1 

.70  1 

386 

1 

291  1 

693 

1 

5301 

1000 

1 

771  1 

479 

1 

361  1 

739 

5631 

1000 

7671 

^  A  1 
.  t)W  1 

1      1  AO 

'7'7  1 
(  f  1 

159  1 

324 

O/tO  1 

^tti  1 

1  oO 

OA  1 

OvO  1 

.50  1 

1  52 

35  1 

104 

741 

156 

1 131 

63 

421 

126 

89  1 

189 

1371 

.40  1 

1  30 

19  1 

60 

41  1 

90 

631 

35 

221 

70 

471 

105 

731 

1 1  1 

oo 

241 

57 

oo  1 

Ir*  1 

1  1  1 
O  1  1 

.20  1 

1  13 

71 

26 

16  1 

39 

25  1 

15 

81 

30 

181 

45 

29  1 

.  10  1 

1  8 

41 

16 

9  1 

24 

14  1 

1 1 

5  1 
1 

22 

12  1 
1 

33 

20  1 

1 

.70 

1 

.60  1 

1 

1  469 

1 

306  1 

734 

1 

4851 

1000 

1 

667  1 

586 

3821 

793 

5221 

1000 

663  1 

.50  1 

1  122 

741 

244 

154  1 

366 

236  1 

155 

941 

310 

196  1 

465 

300  1 

.40  1 

1  55 

31  1 

1 10 

66  1 

165 

1021 

70 

39  1 

140 

841 

210 

130  1 

.30  1 

1  31 

16  1 

62 

35  1 

93 

55  1 

37 

19  1 

74 

421 

1 1 1 

65  1 

.20  1 

1  19 

9  1 

38 

20  1 

57 

321 

24 

111 

48 

25  1 

72 

401 

.  10  1 

1  12 

51 

24 

121 

36 

19  1 

15 

61 

30 

14  1 

1 

45 

23  1 
1 

.60 

1 

.50  1 

1 

1  515 

1 

2841 

757 

1 

4231 

1000 

1 

565  1 

635 

i 

3501 

817 

455  1 

1000 

561  1 

.40  1 

1  127 

641 

254 

135  1 

381 

2071 

157 

79  1 

314 

166  1 

471 

256  1 

.30  1 

1  55 

25  1 

1 10 

541 

165 

85  1 

70 

321 

140 

69  1 

210 

1081 

.20 

1  30 

121 

60 

271 

90 

431 

35 

141 

70 

321 

105 

501 

.  10 

1  17 

6  1 

34 

141 

51 

23  1 

21 

71 
1 

42 

171 

1 

63 

281 
1 

.50 

.40 

1 

1  515 

1 

2321 

757 

3471 1000 

1 

464  1 

635 

286  1 

817 

3721 

1000 

4601 

.30 

1  122 

49  1 

244 

1041 

366 

161  1 

155 

621 

310 

1331 

465 

205  1 

.20 

1  52 

181 

104 

401 

156 

641 

63 

221 

126 

49  1 

189 

77  1 

.  10 

1  24 

71 

48 

16  1 

72 

26  1 
1 

31 

9  1 

1 

62 

21  1 

1 

93 

341 

1 

.40 

.30 

1 

1  469 

1 

1641 

734 

2641 1000 

365  1 

586 

205  1 

793 

2821 

1000 

361  1 

.20 

1  108 

321 

216 

701 

324 

1 10  1 

135 

401 

270 

881 

405 

1371 

.  10 

1  38 

9  1 

76 

21  1 

114 

341 

51 

121 
1 

102 

29  1 

1 

153 

46  1 

1 

.30 

.20 

1 

1  386 

1 

96  1 

693 

181 1 1000 

1 

2671 

479 

1191 

739 

191  1 

1000 

2641 

.  10 

1  78 

15  1 

156 

341 

234 

55  1 

99 

19  1 

1 

198 

431 

1 

297 

70  1 

1 

.20 

.  10 

1 

1  259 

1 

381 

518 

1 

831 

777 

1 

131  1 

321 

471 

642 

1031 

963 

1621 

73 


Table  3^ 


VALUES  OF  n,M  FOR  SELECTING,   VITH  P(CS)   >=  ?*,   A  SUBSET  OF  k  =  10 
BINOMIAL  POPULATIONS  VHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  MB  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


STANDARD  1 
PROPORTIONS! 
UPPER  LOWER 1 

n. 

M 

P*  = 
n 

.90 

M 

n 

1 

1 
1 
1 

M  i 

n 

ri 

PJf:  = 
n 

.95 

M 

n 

n 

1 
1 
1 

1 

M  1 

n  1 

.90 

1 

.80  1 

266 

1 

2281 

532 

1 

4621 

798 

1 

6981 

328 

1 

281  1 

656 

1 

5701 

984 

861  1 

.70  1 

83 

681 

166 

1401 

249 

2131 

100 

821 

200 

1681 

300 

256  1 

.  60  1 

OO  1 

oo  1 

OA 

1 

IMo  1 

O  1 

4M  1 

1  AO 

OO  1 

i  J?  O 

loo 

1  0*7  1 

.50  1 

1  O  1 

OU 

70 

^  1  1 
6  1  1 

o  o 

£^  A 

R  1  1 
o  1  1 

7o  1 

.  40  1 

18 

lo  1 

36 

E?  A 

54 

4o  1 

O  -1 

15  1 

A  O 

42 

OO  1 

63 

50  1 

.30  1 

12 

n  1 

8  1 

O  A 

■t  O  1 

18  1 

O  £. 

36 

<^8  1 

15 

10  1 

O  /X 

30 

OO  1 

22  1 

45 

35  1 

.20  1 

8 

5  1 

16 

1  1  1 

24 

181 

1 1 

71 

22 

16  i 

33 

25  1 

.  10  1 

5 

31 

10 

6  1 

15 

10  1 

7 

41 

14 

9  1 

21 

15  1 

.80 

.70  1 

398 

1 

300  1 

699 

1 

534  1 

1000 

1 

770  1 

49  1 

1 

370  1 

745 

1 

5671 

1000 

1 

767  1 

.60  1 

111 

79  1 

222 

163  1 

333 

249  1 

135 

96  1 

270 

199  1 

405 

303  1 

.50  1 

52 

35  1 

104 

73  1 

156 

1 13  1 

64 

43  1 

128 

901 

192 

139  1 

.  40  1 

30 

19  1 

60 

401 

90 

631 

38 

241 

76 

51  1 

1 14 

801 

.30  1 

20 

12  1 

40 

26  1 

60 

40  1 

24 

14  1 

48 

31  1 

72 

48  1 

.20  1 

13 

7  1 

26 

16  1 

39 

25  1 

15 

8 1 

30 

181 

45 

29  1 

.  10  1 

8 

4  1 

16 

9  1 

24 

14  1 

1  1 

5  1 

22 

121 

33 

201 

.70 

.60  1 

483 

1 

315  1 

74 1 

1 

490  1 

1000 

1 

666  1 

603 

1 

393  1 

801 

1 

5271 

1000 

1 

662  1 

.50  1 

127 

771 

254 

161  1 

381 

246  1 

155 

94  1 

310 

196  1 

465 

3001 

.40  1 

57 

32  1 

1 14 

681 

171 

106  1 

73 

41  1 

146 

881 

219 

136  1 

.30  1 

31 

16  1 

62 

35  1 

93 

55  1 

39 

201 

78 

44  1 

117 

69  1 

.20  1 

20 

9  1 

40 

21  1 

60 

34  1 

24 

1 1  i 

48 

25  1 

72 

40  1 

.  10  1 

12 

5  1 

24 

1 1  1 

36 

19  1 

15 

6  1 

30 

14  1 

45 

23  1 

.60 

.50  1 

526 

1 

2901 

763 

1 

426  1 

1000 

1 

564  1 

655 

1 

361  1 

827 

1 

460  1 

1000 

1 

5601 

.40  1 

131 

66  1 

262 

139  1 

393 

213  1 

163 

821 

326 

1731 

489 

2651 

.30  1 

57 

26  1 

1 14 

56  1 

171 

881 

73 

331 

146 

72  1 

219 

1131 

.20  1 

30 

121 

60 

271 

90 

431 

38 

15  1 

76 

35  1 

1 14 

55  1 

.  10  1 

18 

6  1 

36 

15  1 

54 

241 

21 

71 

42 

17  1 

63 

281 

.50 

.40  1 

526 

1 

237  1 

763 

1 

350  1 

1000 

1 

464  1 

655 

1 

295  I 

827 

377  1 1000 

1 

459  1 

.30  1 

127 

51  1 

254 

109  1 

381 

1681 

155 

621 

310 

1321 

465 

205  1 

.20  1 

52 

181 

104 

40  1 

156 

64  1 

64 

221 

128 

50  1 

192 

781 

.  10  i 

25 

71 

50 

171 

75 

281 

32 

9  1 

64 

221 

96 

35  1 

.40 

.30  1 

483 

1 

169  1 

741 

1 

266  1 

1000 

1 

3641 

603 

1 

211  1 

801 

1 

285  1 

1000 

1 

3601 

.20  1 

1 1 1 

33  1 

222 

721 

333 

1 13  1 

135 

401 

270 

881 

405 

137  1 

.  10  1 

42 

10  1 

84 

231 

126 

381 

51 

121 

102 

281 

153 

46  1 

.30 

.20  1 

398 

1 

99  1 

699 

1 

1821 

1000 

1 

267  1 

491 

1 

1221 

745 

1921 1000 

1 

2631 

.  10  1 

83 

16  1 

166 

36  1 

249 

531 

100 

19  1 

200 

44  1 

300 

701 

.20 

.  10  1 

266 

1 

39  1 

532 

1 

86  1 

798 

1 

134  i 
 1- 

328 

1 

481 

656 

1 

105  1 

984 

1 

165  1 

 1 

74 


Table  35 


VALUES  OF  n.M  FOR  SELECTING,   WITH  P(CS)   >=  P*,   A  SUBSET  OF  k  =  11 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


P*  = 

.90 

1 

P*  = 

.95 

STANDARD  1 

1 

PROPORTIONS! 

- 

1 

UPPER 

LOWER  1 

n 

M 

n 

M 

n 

M  1 

n 

M 

n 

M 

n 

M 

.90 

.80  1 

273 

1 

2341 

546 

1 

4741 

819 

1 

716  1 

335 

1 

2871 

667 

1 

579  1 

1000 

875 

.70  1 

84 

69  1 

168 

1421 

252 

215  1 

100 

821 

200 

1681 

300 

256 

.60  1 

f\Ot  1 

V  V  1 

1 1 

5  1 

1  OiO 

1  V/  u 

Rfl  1 

I J  O  1 

1  o  o 

1  27 

.50  1 

25 

191 

50 

401 

75 

611 

32 

24  1 

64 

51  1 

96 

78 

.40  1 

18 

131 

36 

281 

54 

43  1 

21 

15  1 

42 

321 

63 

50 

OA  1 

12 

81 

24 

181 

36 

281 

15 

10  1 

30 

22  1 

45 

35 

.20  1 

9 

6  1 

18 

131 

27 

20  1 

1 1 

71 

22 

16  1 

33 

25 

.  10  1 

5 

31 

10 

6  1 

15 

10  1 

7 

41 

14 

9  1 
1 

21 

15 

.80 

.70  1 

410 

1 

309  1 

705 

1 

539  1 

1000 

1 

7701 

503 

1 

379  1 

751 

572  1 

1000 

767 

.  O0  1 

1 14 

81  1 

228 

1681 

342 

256  1 

141 

1001 

282 

2081 

423 

316 

.50  1 

55 

371 

1 10 

781 

165 

120  1 

66 

44  1 

132 

93  1 

198 

143 

.40  1 

30 

19  1 

60 

40  1 

90 

63  1 

38 

241 

76 

51  t 

1 14 

80 

.30  1 

O  1 

oo 

1  4i  1 

1  *x  1 

O  1  1 

c  ^ 

.  20  1 

13 

71 

26 

16  1 

39 

25  1 

15 

81 

30 

181 

45 

29 

.  10  1 

9 

4  1 

18 

10  1 

27 

16  1 

1 1 

5  1 
1 

22 

121 

1 

33 

20 

.70 

.  60  1 

RAO 

1 

OOO  1 
Oi^O  1 

<o  i 

1 

AQ^  1 
TJVD  1 

1  AHA 

1 

OOO  1 

D  1  O 

1  1 

t  AAA 

.50  i 

132 

801 

264 

167  1 

396 

256  1 

160 

971 

320 

202  1 

480 

310 

.40  1 

61 

341 

122 

731 

183 

1 13  1 

73 

411 

146 

871 

219 

135 

.30  1 

o  o 

17  1 

DO 

Oo  1 

OA  1 

(  o 

A<dj  1 

! 

lit 

.20  1 

21 

101 

42 

22  1 

63 

35  1 

24 

1  1  1 

48 

25  1 

72 

40 

.  10  1 

12 

5  1 

24 

11  1 

36 

19  1 

15 

6  1 

30 

14  1 

1 

45 

23 

.60 

.50  1 

546 

1 

301  1 

773 

1 

4321 

1000 

1 

564  1 

673 

1 

371  1 

836 

465  1 

1000 

559 

.40  1 

135 

681 

270 

1431 

405 

220  1 

167 

841 

334 

177  1 

501 

272 

.30  1 

61 

281 

122 

601 

183 

94  1 

73 

331 

146 

721 

219 

1 12 

.20  1 

30 

121 

60 

27  1 

90 

431 

38 

15  1 

76 

34  1 

1 14 

55 

.  10  1 

18 

6  1 

36 

15  1 

54 

24  1 

21 

71 

1 

42 

171 

1 

63 

28 

.50 

.40  1 

546 

1 

246  1 

773 

1 

3541 

1000 

1 

4631 

673 

3031 

836 

3801 

1000 

459 

.30  1 

132 

531 

264 

1131 

396 

175  1 

160 

641 

320 

137  1 

480 

212 

.20  1 

55 

19  1 

110 

431 

165 

671 

66 

231 

132 

51  1 

198 

81 

.  10  1 

25 

71 

50 

171 

75 

271 

32 

9  1 
1 

64 

22  1 

1 

96 

35 

.40 

.30  1 

503 

1 

176  1 

751 

1 

269  1 

1000 

1 

3641 

615 

215  1 

807 

2871 

1000 

360 

.20  1 

1 14 

34  1 

228 

741 

342 

1 16  1 

141 

421 

282 

92  1 

423 

143 

.  10  1 

42 

10  1 

84 

23  1 

126 

381 

1 

51 

121 

1 

102 

281 

1 

153 

46 

.30 

.20  1 

410 

1 

102  1 

705 

1 

1831 

1000 

266  1 

503 

125  1 

751 

1931 

1000 

263 

.  10  1 

84 

16  1 

168 

37  1 

252 

59  1 

100 

19  1 

1 

200 

441 

1 

300 

70 

.20 

.  10  1 

273 

1 

40  1 

546 

1 

881 

819 

1 

1371 

335 

49  1 

667 

1071 

1000 

168 

75 


Table  36 


VALUES  OF  n,M  FOR  SELECTING,   WITH  P(CS)   >=  ?*,   A  SUBSET  OF  k  =  12 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


STANDARD  1 
PROPORTIONS! 
UPPER  LOWER 1 

n 

M 

P*  = 
n 

.  90 
M 

n 

1 
1 
1 

M  1 

n 

M 

P*  = 
n 

.  95 
M 

n 

1 
1 
1 

M  1 

.  90 

1 

.  80  1 

286 

1 

245  1 

572 

1 

497  1 

858 

1 

75  1  1 

348 

1 

2981 

674 

1 

585  i 

1000 

1 

874  1 

.  70  1 

84 

69  i 

168 

141  1 

252 

215  1 

105 

86  1 

210 

177  1 

315 

269  1 

.  60  1 

43 

34  1 

86 

70  1 

129 

107  1 

55 

43  1 

1 10 

90  1 

165 

138  1 

.50  1 

28 

21  1 

56 

'45  1 

84 

69  1 

32 

24  1 

64 

51  1 

96 

781 

.40  1 

18 

13  1 

36 

28 1 

54 

43  1 

21 

15  1 

42 

321 

63 

50  1 

.  30  1 

13 

■  9  1 

26 

19  1 

39 

30  1 

15 

10  1 

30 

221 

45 

35  1 

.  20  1 

9 

6  1 

18 

13  1 

27 

20  1 

1 1 

7  1 

22 

15  1 

33 

25  1 

.  10  1 

5 

3  1 

10 

6  1 

15 

10  1 

7 

4  1 

14 

9  1 

21 

15  1 

.80 

.70  1 

422 

1 

3181 

711 

1 

5431 

1000 

1 

769  1 

515 

1 

3881 

757 

1 

576  1 

1000 

1 

766  1 

.60  1 

118 

84  1 

236 

174  1 

354 

265  1 

142 

10  I  1 

284 

209  1 

426 

319  1 

.  50  1 

55 

37  1 

1  10 

78 1 

165 

119  1 

69 

46  1 

138 

981 

207 

150  1 

.40  1 

32 

20  1 

64 

43  1 

96 

67  1 

40 

25  ! 

80 

54  1 

120 

84  1 

.30  1 

21 

12  1 

42 

27  1 

63 

42  1 

24 

14  1 

48 

31  1 

72 

481 

.20  1 

13 

7  1 

26 

16  1 

39 

25  1 

15 

81 

30 

181 

45 

281 

.  10  1 

9 

4  1 
1 

18 

10  1 

27 

16  1 

1 1 

5  1 

22 

121 

33 

20  1 

.70 

.60  1 

517 

337  1 

758 

1 

500  1 

1000 

1 

665  1 

629 

1 

410  1 

814 

1 

535  1 

1000 

1 

66  1  1 

.50  1 

135 

82  1 

270 

171  1 

405 

261  1 

165 

1001 

330 

209  1 

495 

319  1 

.40  1 

62 

35  1 

124 

74  1 

186 

1  15  1 

75 

42  1 

150 

901 

225 

139  1 

.30  1 

33 

17  1 

66 

37  1 

99 

581 

41 

21  1 

82 

46  1 

123 

72  1 

.20  1 

21 

10  1 

42 

221 

63 

35  1 

24 

1 1  1 

48 

25  1 

72 

40  1 

.  10  1 

13 

5  1 

26 

12  1 

39 

201 

15 

6  1 

30 

14  1 
1 

45 

23  1 

.60 

.50  1 

564 

1 

311  1 

782 

1 

436  1 

1000 

1 

563  1 

686 

1 

3781 

843 

4681 1000 

1 

559  1 

.40  1 

139 

70  1 

278 

1471 

417 

226  1 

169 

85  1 

338 

179  1 

507 

275  1 

.30  1 

62 

281 

124 

61  1 

186 

96  1 

75 

341 

150 

741 

225 

1 16  1 

.20  1 

32 

131 

64 

29  1 

96 

46  1 

40 

16  1 

80 

36  1 

120 

581 

.  10  1 

18 

6  1 

36 

15  1 

54 

24  1 

21 

7  1 

42 

17  1 

63 

27  1 

.50 

.40  1 

564 

1 

254  1 

782 

1 

3581 

1000 

1 

4621 

686 

1 

309  1 

843 

3831 1000 

1 

4581 

.30  1 

135 

54  1 

270 

1 16  1 

405 

179  1 

165 

66  1 

330 

141  1 

495 

2181 

.20  1 

55 

19  1 

1 10 

43  1 

165 

67  1 

69 

24  1 

138 

54  1 

207 

85  1 

.  10  1 

28 

81 

56 

19  1 

84 

31  1 

32 

9  1 

64 

221 

96 

35  1 

.40 

.30  1 

517 

1 

181  1 

758 

1 

271  1 

1000 

1 

3631 

629 

1 

2201 

814 

1 

289  1 

1000 

1 

359  1 

.20  1 

1 18 

35  1 

236 

77  1 

354 

1201 

142 

421 

284 

921 

426 

1441 

.  10  1 

43 

101 

86 

241 

129 

39  1 

55 

131 

1 10 

31  1 

165 

501 

.30 

.20  1 

422 

1 

105  1 

711 

1 

185  1 

1000 

1 

266  1 

515 

1 

1281 

757 

194 1 1000 

1 

2621 

.  10  1 

84 

16  1 

168 

37  1 

252 

59  1 

105 

20  1 

210 

46  1 

315 

74  1 

I 

.20 

.  10  1 

286 

1 

421 

572 

1 

921 

858 

1 

144  1 
 1- 

348 

1 

511 

674 

1 

1081 

1000 

1671 

 1 

76 


Table  37 


VALUES  OF  n,M  FOR  SELECTING,  WITH  P(CS)   >=  F*,   A  SUBSET  OF  k  =  13 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


STANDARD  1 
PROPORTIONS  1 
UPPER  LOWER 1 

n 

M 

P*  = 
n 

.90 

M 

n 

1 
1 
1 

M  1 

n 

M 

P*  = 
n 

.95 

M 

n 

1 
1 
1 

M  1 

.90 

1 

.80  1 

293 

1 

251  1 

586 

1 

509  1 

879 

1 

769  1 

355 

1 

304  1 

677 

1 

588  1 

1000 

1 

874  1 

.70  1 

89 

731 

178 

1501 

267 

2281 

105 

86  1 

210 

177  1 

315 

269  1 

.60  1 

46 

36  1 

92 

75  1 

138 

1 15  1 

55 

43  1 

1 10 

90  1 

165 

138  1 

28 

21  1 

56 

44  1 

84 

681 

32 

24  1 

64 

5  1  1 

96 

78  1 

.40  1 

18 

13  1 

36 

28  1 

54 

43  1 

21 

15  1 

42 

32  1 

63 

50  1 

.30  1 

13 

^  1 

26 

19  1 

39 

30  1 

15 

10  1 

30 

22  1 

45 

34  1 

.20  1 

10 

6  1 

20 

14  1 

30 

23  1 

1 1 

7  1 

22 

15  1 

33 

24  1 

.  10  1 

7 

41 

14 

9  1 

21 

15  1 

7 

41 

14 

9  1 

21 

15  1 

.80 

,70  1 

434 

1 

327  1 

717 

1 

547  1 

1000 

1 

769  1 

527 

1 

397  1 

763 

1 

580  1 

1000 

1 

766  1 

.60  1 

1 18 

84  1 

236 

174  1 

354 

265  1 

145 

103  1 

290 

213  1 

435 

325  1 

.50  1 

58 

39  1 

1 16 

82  1 

174 

126  1 

69 

46  1 

138 

97  1 

207 

150  1 

1 

35 

221 

70 

481 

105 

741 

40 

25  1 

80 

54  1 

120 

841 

.30  1 

22 

13  1 

44 

281 

66 

45  1 

24 

14  1 

48 

31  1 

72 

48  1 

.20  1 

13 

7  1 

26 

16  1 

39 

25  1 

17 

9  1 

34 

21  1 

5  1 

33  1 

10  1 

10 

5  f 

20 

1 1  1 

30 

181 

1 1 

5  1 

1 

22 

12  1 

1 

33 

20  1 
1 

.70 

.60  1 

526 

1 

343  1 

763 

1 

503  1 

1000 

1 

665  1 

646 

421  1 

823 

541  1 

1000 

66  1  1 

.50  1 

140 

85  1 

280 

177  1 

420 

271  1 

170 

1031 

340 

215  1 

510 

329  1 

.40  1 

64 

36  1 

128 

771 

192 

119  1 

75 

421 

150 

90  1 

225 

139  1 

.30  1 

33 

17  t 

66 

37  1 

99 

581 

41 

21  1 

82 

46  1 

123 

72  1 

.20  1 

22 

10  1 

44 

231 

66 

371 

24 

1 1  1 

48 

25  1 

72 

40  1 

.  10  1 

13 

5  1 

26 

121 

39 

201 

15 

6  1 

1 

30 

141 

1 

45 

231 
1 

.60 

.50  1 

575 

1 

3171 

787 

1 

439  1 

1000 

1 

5631 

704 

3881 

852 

473  1 

1000 

559  1 

.40  1 

143 

721 

286 

1521 

429 

233  1 

173 

871 

346 

1831 

519 

282  1 

.30  1 

64 

29  1 

128 

631 

192 

99  1 

75 

341 

150 

741 

225 

1 15  1 

.20  1 

35 

141 

70 

321 

105 

511 

40 

16  1 

80 

36  1 

120 

581 

.  10  1 

18 

6  1 

36 

14  1 

54 

241 

21 

71 
1 

42 

171 

1 

63 

271 

1 

.50 

.40  1 

575 

1 

259  1 

787 

1 

3601 

1000 

1 

4621 

704 

3171 

852 

3871 

1000 

4581 

.30  1 

140 

56  1 

280 

1201 

420 

185  1 

170 

681 

340 

146  1 

510 

225  1 

.20  1 

58 

201 

1 16 

45  1 

174 

71  1 

69 

241 

138 

531 

207 

84  1 
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28 

81 

56 
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84 

31  1 

1 

32 
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1 

64 

21  1 

1 

96 
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1 

.40 

.30  1 

526 

1 

1841 

763 

1 

2731 

1000 

3631 

646 

226  1 

823 

2921 

1000 

359  1 

.20  1 

118 

35  1 

236 

76  1 

354 

1201 

145 

431 

290 

941 

435 

1471 

.  10  1 

46 

11  1 

92 

26  1 

138 

421 

55 

131 
1 

1 10 

31  1 
1 

165 

501 

1 

.30 

.20  1 

434 

1 

1081 

717 

1 

186  1 

1000 

1 

265  1 

527 

131  1 

763 

196  1 

1000 

2621 

.  10  1 

89 

171 

178 

391 

267 

621 

1 

105 

201 

1 

210 

46  1 

1 

315 

731 

1 

.20 

.  10  1 

293 

1 

431 

586 

1 

941 

879 

1481 

355 

521 

677 

1081 

1000 

1671 

77 


Table  38 


VALUES  OF  n,M  FOR  SELECTING,   WITH  P(CS)  >=  P*,   A  SUBSET  OF  k  =  14 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  ALL  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF   'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE,  SELECJED) 


P*  = 

.90 

1 

P*  = 

.95 

1 
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41 
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14 
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21 
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1 
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80 

54  1 
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44 

281 

66 
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26 
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.20  1 

1  O 

'7  1 

1  O  1 
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1 
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.50  1 

1 

OO  1 
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^(11 
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1  UO  i 

O'tW 
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1  O  1 

O  Iv 
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.  4v>  1 

64 

36  1 

128 

77  1 

192 

1 19  1 

77 

43  1 

154 

92  1 

231 

143  1 

,30  1 

35 

181 

70 

39  1 

105 

62  1 

41 

21  ! 

82 

46  1 

123 

72  1 

.20  I 

22 

10  1 

44 

23  1 

66 

37  1 

26 

12  1 

52 

271 

78 

43  1 

.  10  1 

1  O 

O  1 

1 

1  o 

D  1 

Ov 

OQ  1 

.60 

.50  1 

586 

1 

3231 

793 

1 

4421 

1000 

1 

5621 

715 

1 

394  1 

857 

1 

475  1 

1000 

1 

5581 

.40  1 

145 

73  1 

290 

154  1 

435 

236  1 

177 

89  1 

354 

187  1 

531 

2881 

.30  i 

64 

29  1 

128 

631 

192 

99  1 

77 

35  1 

154 

76  1 

231 

1131 

.20  1 

35 

141 

70 

32  1 

105 

5  1  1 

40 

16  1 

80 

36  1 

120 

57  1 

.  10  1 

18 

6  1 

36 

141 

54 

24  1 

21 

71 

42 

17  1 

63 

271 

.50 

.40  1 

586 

1 

2641 

793 

1 

3621 

1000 

1 

462  1 

715 

1 

3221 

857 

1 

389  1 

1000 

1 

4581 

.30  1 

140 

56  1 

280 

120  1 

42^ 

185  1 

170 

681 

340 

145  1 

510 

225  1 

.20  1 

58 

20  1 

1 16 

45  1 

174 

71  1 

72 

25  1 

144 

56  1 

216 

881 

.  10  1 

28 

81 

56 

19  i 

84 

31  1 

32 

9  1 

64 

21  1 

96 

35  1 

.40 

.30  i 

543 

1 

1901 

771 

1 

275  1 

1000 

1 

3621 

655 

1 

229  1 

827 

1 

2931 

1000 

1 

359  1 

.20  1 

121 

36  1 

242 

781 

363 

1231 

149 

44  1 

298 

971 

447 

151  1 

.  10  1 

46 

11  1 

92 

26  1 

138 

41  I 

55 

131 

1 10 

30  1 

165 

49  1 

.30 

.20  1 

442 

1 

1101 

721 

1 

1871 

1000 

1 

265  i 

535 

1 

1331 

767 

1 

197  1 

1000 

1 

2621 

.  10  1 

89 

171 

178 

39  1 

267 

621 

1 10 

211 

220 

481 

330 

771 

.20 

.  10  1 

299 

1 

44  1 

598 

1 

96  1 

897 

1 

151  1 
 1- 

362 

1 

531 

681 

1 

109  1 

1000 

1 

1671 
 1 

78 


Table  39 


VALUES  OF  n,M  FOR  SELECTING,  WITH  P(CS)   >=  P*,   A  SUBSET  OF  k  =  15 
BINOMIAL  POPULATIONS  WHICH  CONTAINS  A1.L  POPULATIONS  BETTER  THAN  AN 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.  OF  'SUCCESSES'   >=  M  OUT  OF  n  TRIALS  ARE  SELECTED) 


P*  = 

.90 

1 

P*  = 

.95 

1 

STANDARD  1 

1 
1 

1 
1 

PROPORTIONS! 

1 
1 

1 
1 

UPPER 

LOWER 1 

n 

ri 

n 

n 

n 

M  1 

n  \ 

n 

n 

n 

Pi 

n 

Fl  1 

.  90 

.  80  1 

300 

1 

2571 

600 

1 

521  1 

900 

1 

7871 

368 

1 

3151 

684 

594  1 

1000 

1 

8741 

.  70  i 

94 

771 

188 

1581 

282 

241  1 

111 

91  1 

222 

187  1 

333 

284! 

.  60  1 

1 

o7  1 

94 

77  1 

t  A  t 

14 1 

1  lo  1 

06 

>f  A  1 

44  1 

1  1<2 

9  1  1 

loo 

14w  1 

.50  1 

28 

2  1  1 

56 

44  I 

84 

68  1 

35 

26  1 

70 

55  1 

0  e?  1 

o5  1 

.40  1 

18 

13  1 

36 

27  1 

54 

43  1 

24 

17  1 

48 

37  1 

72 

57  1 

.  30  1 

13 

9  1 

26 

19  1 

39 

30  1 

16 

1 1  1 

32 

24  1 

48 

37  1 

.  20  1 

10 

6  1 

20 

141 

30 

221 

1 1 

71 

22 

15  1 

33 

24  1 

.  10  1 

7 

41 

14 

91 

21 

15  1 

7 

41 

14 

9  1 

1 

21 

15  1 

.80 

.70  1 

454 

1 

342  1 

727 

1 

555  1 

1000 

1 

768  1 

547 

1 

412  1 

773 

588  1 

1000 

1 

765  1 

.60  1 

125 

89  1 

250 

184  1 

375 

280  1 

152 

108 1 

304 

224  1 

456 

34 1  1 

.  50  1 

60 

40  1 

120 

85  1 

180 

1301 

72 

481 

144 

102  1 

216 

156  1 

.40  1 

35 

221 

70 

471 

105 

731 

40 

25  1 

80 

54  1 

120 

841 

.30  1 

22 

13  1 

44 

28 1 

66 

44  1 

26 

15  1 

52 

33  1 

78 

52  1 

.20  1 

15 

81 

30 

181 

45 

29  1 

17 

9  1 

34 

20  1 

5  1 

33  1 

.  10  1 

10 

5  1 

20 

1 1  1 

30 

181 

1 1 

5  1 

22 

121 

1 

33 

201 

1 

.70 

.60  1 

549 

1 

3581 

774 

1 

5 10  1 

1000 

1 

664  1 

669 

1 

436  1 

834 

548 ! 

1000 

660  1 

.  50  1 

145 

881 

290 

184  1 

435 

281  1 

175 

106  1 

350 

221  1 

525 

339  1 

.40  1 

64 

36  1 

128 

77  1 

192 

119  1 

80 

45  1 

160 

96  1 

240 

1481 

.30  1 

35 

181 

70 

39  1 

105 

621 

43 

221 

86 

481 

129 

76  1 

.20  1 

22 

10  1 

44 

23  1 

66 

37  1 

26 

12  1 

52 

27  1 

78 

A  0  1 

43  1 

.  10  1 

13 

5  1 

26 

121 

39 

201 

16 

6  1 
1 

32 

15  1 

48 

25  1 

1 

.60 

.50  1 

597 

1 

329  1 

798 

1 

445  i 

1000 

i 

562  i 

726 

4001 

863 

479  1 

1000 

5581 

.40  1 

149 

75  1 

298 

158i 

447 

2431 

179 

901 

358 

1901 

537 

291  1 

.30  1 

64 

29  1 

128 

631 

192 

981 

80 

36  1 

160 

79  1 

240 

1231 

.20  1 

35 

141 

70 

321 

105 

51  1 

40 

16  1 

80 

36  1 

120 

57! 

.  10  1 

18 

6  1 

36 

14  1 

54 

231 

24 

81 

48 

201 

1 

72 

321 
1 

.50 

.40  1 

597 

1 

269  1 

798 

1 

3641 

1000 

1 

461  1 

726 

1 

3271 

863 

3921 

1000 

4571 

.30  1 

145 

581 

290 

1241 

435 

1921 

175 

701 

350 

1501 

525 

2321 

.20  1 

60 

21  1 

120 

47  1 

180 

741 

72 

251 

144 

56  1 

216 

881 

.  10  1 

28 

81 

56 

19  1 

84 

311 

1 

35 

101 

1 

70 

241 

1 

105 

39  1 
1 

.40 

.30  1 

549 

1 

1921 

774 

1 

276  1 

1000 

3621 

669 

2341 

834 

296  1 

1000 

3581 

.20  1 

125 

371 
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81  1 
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1271 

152 
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304 

99  1 
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1541 
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94 
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141 
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311 
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41  1 

282 

66  1 
1 
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211 

1 

222 

49  1 

1 

333 

781 
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.20 

.  10  1 

300 

1 

441 
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1 

96  1 
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151  1 

368 

541 

684 

109  1 

1000 

166  1 

79 


Table  ho 


VALUES  OF  n,M  FOR  SELECTING,   WITH  P(CS)   >=  A  SUBSET  OF  k  =  16 

BIWOMIAL  POPULATIONS  ¥HICH  CONTAINS  ALL  POPULATIONS  BETTER  THi\N  AW 
UPPER  STANDARD  AND  NO  POPULATIONS  WORSE  THAN  A  LOWER  STANDARD 
(POPULATIONS  WITH  NO.   OF   'SUCGE-SSES'   >=  M  OUT  OF  n  TRIALS  ARE,  SELECTED) 
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